{
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "WClxySHIzi9G"
      },
      "source": [
        "![Cohere-Logo-Color-RGB.png]()"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "LplM9PSe8djM"
      },
      "source": [
        "# Evaluating summarization\n",
        "\n",
        "In this cookbook, we will be demonstrating an approach we use for evaluating summarization tasks using LLM evaluation.\n",
        "\n",
        "### Table of contents:\n",
        "1. [Get started](#start)\n",
        "2. [Construct the evaluation dataset](#dataset)\n",
        "3. [Build the evaluation framework](#eval-framework)\n",
        "5. [Run evaluations](#run-evals)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6SfEhd3O74--"
      },
      "source": [
        "<a id=\"start\"></a>\n",
        "<a name=\"start\"></a>\n",
        "# Get Started\n",
        "\n",
        "You'll need a Cohere API key to run this notebook. If you don't have a key, head to https://cohere.com/ to generate your key."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "H2TfKiPM3a6f"
      },
      "outputs": [],
      "source": [
        "# %%capture\n",
        "!pip install \"cohere<5\" datasets --quiet"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "J1HzGqnY74bj"
      },
      "outputs": [],
      "source": [
        "import json\n",
        "import random\n",
        "import re\n",
        "from typing import List, Optional\n",
        "\n",
        "import cohere\n",
        "from getpass import getpass\n",
        "from datasets import load_dataset\n",
        "import pandas as pd\n",
        "\n",
        "# Set up Cohere client\n",
        "co_api_key = getpass(\"Enter your Cohere API key: \")\n",
        "co_model = \"command-r\"\n",
        "co = cohere.Client(api_key=co_api_key)"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dEN21GS913wo"
      },
      "source": [
        "As test data, we'll use transcripts from the [QMSum dataset](https://github.com/Yale-LILY/QMSum). Note that in addition to the transcripts, this dataset also contains reference summaries -- we will use only the transcripts as our approach is reference-free."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "1JfU70r113wo"
      },
      "outputs": [],
      "source": [
        "qmsum = load_dataset(\"MocktaiLEngineer/qmsum-processed\", split=\"validation\")\n",
        "transcripts = [x for x in qmsum[\"meeting_transcript\"] if x is not None]"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "3U58iKz213wo"
      },
      "source": [
        "<a id=\"dataset\"></a>\n",
        "<a name=\"dataset\"></a>\n",
        "# Construct the evaluation dataset\n",
        "\n",
        "We are interested in evaluating summarization in real-world, enterprise use cases, which typically have two distinguishing features as compared to academic summarization benchmarks:\n",
        "- Enterprise use cases often focus on specific summarization objectives, e.g. \"summarize action items\".\n",
        "- Enterprise use cases often feature specific instruction constraints, e.g. \"summarize in bullets with each bullet under 20 words\".\n",
        "\n",
        "Therefore, we must first create a dataset that contains diverse summarization prompts. We will do this programmatically by building prompts from their components, as defined below:\n",
        "- Prompt = text (e.g. transcript to be summarized) + instruction\n",
        "- Instruction = instruction objective (e.g. \"summarize action items\") + modifiers\n",
        "- Modifiers = format/length modifiers (e.g. \"use bullets\") + style/tone modifiers (e.g. \"do not mention names\") + ...\n",
        "\n",
        "First, we define the prompt that combines the text and instructions. Here, we use a very basic prompt:"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "2Bj2I0If13wp"
      },
      "outputs": [],
      "source": [
        "prompt_template = \"\"\"## meeting transcript\n",
        "{transcript}\n",
        "\n",
        "## instructions\n",
        "{instructions}\"\"\""
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vc33XZLL13wp"
      },
      "source": [
        "Next, we build the instructions. Because each instruction may have a different objective and modifiers, we track them using metadata. This will later be required for evaluation (i.e. to know what the prompt is asking)."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "4uMI5VXm13wp"
      },
      "outputs": [],
      "source": [
        "# In this cookbook, we will only consider the following 2 objectives and 4 format/length modifiers\n",
        "\n",
        "instruction_objectives = {\n",
        "    \"general_summarization\": \"Summarize the meeting based on the transcript.\",\n",
        "    \"action_items\": \"What are the follow-up items based on the meeting transcript?\",\n",
        "}\n",
        "\n",
        "# Note that not all format/length modifiers are compatible with all objectives. For example, action\n",
        "# items are best suited for bullet format.\n",
        "format_length_modifiers = {\n",
        "    \"paragraphs_short\": {\n",
        "        \"text\": \"In paragraph form, output your response. Use at least 10 words and at most 50 words in total.\",\n",
        "        \"objectives\": [\"general_summarization\"],\n",
        "        \"eval_metadata\": {\n",
        "            \"format\": \"paragraphs\",\n",
        "            \"min_length\": 10,\n",
        "            \"max_length\": 50,\n",
        "        },\n",
        "    },\n",
        "    \"paragraphs_medium\": {\n",
        "        \"text\": \"Return the answer in the form of paragraphs. Make sure your answer is between 50 and 200 words long.\",\n",
        "        \"objectives\": [\"general_summarization\"],\n",
        "        \"eval_metadata\": {\n",
        "            \"format\": \"paragraphs\",\n",
        "            \"min_length\": 50,\n",
        "            \"max_length\": 200,\n",
        "        },\n",
        "    },\n",
        "    \"bullets_short_3\": {\n",
        "        \"text\": \"Format your answer in the form of bullets. Use exactly 3 bullets. Each bullet should be at least 10 words and at most 20 words.\",\n",
        "        \"objectives\": [\"general_summarization\", \"action_items\"],\n",
        "        \"eval_metadata\": {\n",
        "            \"format\": \"bullets\",\n",
        "            \"number\": 3,\n",
        "            \"min_length\": 10,\n",
        "            \"max_length\": 20,\n",
        "        },\n",
        "    },\n",
        "    \"bullets_medium_2\": {\n",
        "        \"text\": \"In bullets, output your response. Make sure to use exactly 2 bullets. Make sure each bullet is between 20 and 80 words long.\",\n",
        "        \"objectives\": [\"general_summarization\", \"action_items\"],\n",
        "        \"eval_metadata\": {\n",
        "            \"format\": \"bullets\",\n",
        "            \"number\": 2,\n",
        "            \"min_length\": 20,\n",
        "            \"max_length\": 80,\n",
        "        },\n",
        "    },\n",
        "}"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u0dqBqxO13wq"
      },
      "source": [
        "Let's combine the objectives and format/length modifiers to finish building the instructions."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "qMTxDOL713wq",
        "outputId": "e854d1bf-0ea7-41cd-b385-98294d19472e"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "[\n",
            "    {\n",
            "        \"instruction\": \"Summarize the meeting based on the transcript. In paragraph form, output your response. Use at least 10 words and at most 50 words in total.\",\n",
            "        \"eval_metadata\": {\n",
            "            \"format\": \"paragraphs\",\n",
            "            \"min_length\": 10,\n",
            "            \"max_length\": 50\n",
            "        },\n",
            "        \"objective\": \"general_summarization\"\n",
            "    },\n",
            "    {\n",
            "        \"instruction\": \"Summarize the meeting based on the transcript. Return the answer in the form of paragraphs. Make sure your answer is between 50 and 200 words long.\",\n",
            "        \"eval_metadata\": {\n",
            "            \"format\": \"paragraphs\",\n",
            "            \"min_length\": 50,\n",
            "            \"max_length\": 200\n",
            "        },\n",
            "        \"objective\": \"general_summarization\"\n",
            "    }\n",
            "]\n"
          ]
        }
      ],
      "source": [
        "instructions = []\n",
        "for obj_name, obj_text in instruction_objectives.items():\n",
        "    for mod_data in format_length_modifiers.values():\n",
        "        for mod_obj in mod_data[\"objectives\"]:\n",
        "            if mod_obj == obj_name:\n",
        "                instruction = {\n",
        "                        \"instruction\": f\"{obj_text} {mod_data['text']}\",\n",
        "                        \"eval_metadata\": mod_data[\"eval_metadata\"],\n",
        "                        \"objective\": obj_name,\n",
        "                    }\n",
        "                instructions.append(instruction)\n",
        "\n",
        "# View the first two instructions\n",
        "print(json.dumps(instructions[:2], indent=4))"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "1MeNfW-m13wr"
      },
      "source": [
        "Finally, let's build the final prompts by semi-randomly pairing the instructions with transcripts from the QMSum dataset."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "LYTMTFAm13wr"
      },
      "outputs": [],
      "source": [
        "data = pd.DataFrame(instructions)\n",
        "\n",
        "# Randomly shuffle the top 25% of transcripts by length, then assign them to the data\n",
        "transcripts = sorted(transcripts, key=lambda x: len(x), reverse=True)[:int(len(transcripts) * 0.25)]\n",
        "random.seed(42)\n",
        "random.shuffle(transcripts)\n",
        "data[\"transcript\"] = transcripts[:len(data)]\n",
        "\n",
        "# Build the prompt\n",
        "data[\"prompt\"] = data.apply(lambda x: prompt_template.format(transcript=x[\"transcript\"], instructions=x[\"instruction\"]), axis=1)"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "igP0-oHN13wr"
      },
      "outputs": [],
      "source": [
        "# (Optional) Let's also check the token lengths of our prompts.\n",
        "data[\"transcript_token_len\"] = [len(x) for x in co.batch_tokenize(data[\"transcript\"].tolist(), model=co_model)]"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "CbPVxiuV13wr",
        "outputId": "bc9e9cf6-31c0-4404-f43f-fdfb5679cd56"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "## meeting transcript\n",
            "PhD F: As opposed to the rest of us \n",
            "PhD D: Well comment OK I I remind that me my first objective eh in the project is to to study difference parameters to to find a a good solution to detect eh the overlapping zone in eh speech recorded But eh tsk comment ehhh comment In that way comment I I I begin to to study and to analyze the ehn the recorded speech eh the different session to to find and to locate and to mark eh the the different overlapping zone And eh so eh I was eh I am transcribing the the first session and I I have found eh eh one thousand acoustic events eh besides the overlapping zones eh I I I mean the eh breaths eh aspiration eh eh talk eh eh clap eh comment I do not know what is the different names eh you use to to name the the pause n speech \n",
            "Grad G: Oh I do not think we ve been doing it at that level of detail So \n",
            "PhD D: Eh I I I do I do not need to to to mmm to m to label the the different acoustic but I prefer because eh I would like to to study if eh I I will find eh eh a good eh parameters eh to detect overlapping I would like to to to test these parameters eh with the another eh eh acoustic events to nnn to eh to find what is the ehm the false eh the false eh hypothesis eh nnn which eh are produced when we use the the ehm this eh parameter eh I mean pitch eh eh difference eh feature \n",
            "PhD A: You know I think some of these that are the nonspeech overlapping events may be difficult even for humans to tell that there s two there I mean if it s a tapping sound you would not necessarily or you know something like that it would be it might be hard to know that it was two separate events \n",
            "Grad G: Well You were not talking about just overlaps were you ? You were just talking about acoustic events \n",
            "PhD D: I I I I t I t I talk eh about eh acoustic events in general but eh my my objective eh will be eh to study eh overlapping zone Eh ? comment n Eh in twelve minutes I found eh eh one thousand acoustic events \n",
            "Professor E: How many overlaps were there in it ? No no how many of them were the overlaps of speech though ? \n",
            "PhD D: How many ? Eh almost eh three hundred eh in one session in five eh in forty five minutes Alm Three hundred overlapping zone With the overlapping zone overlapping speech speech what eh different duration \n",
            "Postdoc B: Does this ? So if you had an overlap involving three people how many times was that counted ? \n",
            "PhD D: three people two people Eh I would like to consider eh one people with difference noise eh in the background be \n",
            "Professor E: No no but I think what she s asking is pause if at some particular for some particular stretch you had three people talking instead of two did you call that one event ? \n",
            "PhD D: Oh Oh I consider one event eh for th for that eh for all the zone This th I I I con I consider I consider eh an acoustic event the overlapping zone the period where three speaker or eh are talking together \n",
            "Grad G: So let s say me and Jane are talking at the same time and then Liz starts talking also over all of us How many events would that be ? \n",
            "PhD D: So I do not understand \n",
            "Grad G: So two people are talking comment and then a third person starts talking Is there an event right here ? \n",
            "PhD D: Eh no No no For me is the overlapping zone because because you you have s you have more one eh more one voice eh eh produced in a in in a moment \n",
            "Grad G: So i if two or more people are talking \n",
            "Professor E: OK So I think We just wanted to understand how you are defining it So then in the region between since there there is some continuous region in between regions where there is only one person speaking And one contiguous region like that you are calling an event Is it Are you calling the beginning or the end of it the event or are you calling the entire length of it the event ? \n",
            "PhD D: I consider the the nnn the nnn nnn eh the entirety eh eh all all the time there were the voice has overlapped This is the idea But eh I I do not distinguish between the the numbers of eh speaker I m not considering eh the the ehm eh the fact of eh eh for example what did you say ? Eh at first eh eh two talkers are eh speaking and eh eh a third person eh join to to that For me it s eh it s eh all overlap zone with eh several numbers of speakers is eh eh the same acoustic event Wi but without any mark between the zone of the overlapping zone with two speakers eh speaking together and the zone with the three speakers \n",
            "Postdoc B: That would j just be one \n",
            "PhD D: Eh with eh a beginning mark and the ending mark Because eh for me is the is the zone with eh some kind of eh distortion the spectral I do not mind By the moment by the moment \n",
            "Grad G: Well but But you could imagine that three people talking has a different spectral characteristic than two \n",
            "PhD D: I I do not but eh but eh I have to study comment What will happen in a general way \n",
            "Grad G: So You had to start somewhere \n",
            "PhD C: So there s a lot of overlap \n",
            "PhD D: I I do not know what eh will will happen with the \n",
            "Grad G: That s a lot of overlap \n",
            "Professor E: So again that s that s three three hundred in forty five minutes that are that are speakers just speakers \n",
            "Postdoc B: But a a a th \n",
            "Professor E: So that s about eight per minute \n",
            "Postdoc B: But a thousand events in twelve minutes that s \n",
            "PhD C: But that can include taps \n",
            "Postdoc B: Well but a thousand taps in eight minutes is a l in twelve minutes is a lot \n",
            "PhD D: I I con I consider I consider acoustic events eh the silent too \n",
            "Grad G: Silence starting or silence ending \n",
            "PhD D: silent ground to bec to detect eh because I consider acoustic event all the things are not eh speech In ge in in in a general point of view \n",
            "Professor E: OK so how many of those thousand were silence ? \n",
            "PhD F: Not speech not speech or too much speech \n",
            "Professor E: Right So how many of those thousand were silence silent sections ? \n",
            "PhD D: silent I I I I do not I I have not the eh I I would like to to do a stylistic study\n",
            "\n",
            "## instructions\n",
            "Summarize the meeting based on the transcript. In paragraph form, output your response. Use at least 10 words and at most 50 words in total.\n"
          ]
        }
      ],
      "source": [
        "# Let's examine one of the prompts\n",
        "print(data[\"prompt\"][0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "tDamVDyP13wr"
      },
      "source": [
        "<a id=\"eval-framework\"></a>\n",
        "<a name=\"eval-framework\"></a>\n",
        "# Build the evaluation framework\n",
        "\n",
        "We now setup the tools we will use for evaluation.\n",
        "\n",
        "We use three criteria that are graded using LLMs:\n",
        "- Completeness: checks if the summary includes all the important information from the original text that it should include\n",
        "- Correctness: checks if there are any hallucinations or factual inaccuracies in the summary\n",
        "- Conciseness: checks if the summary includes any unnecessary information or wordiness\n",
        "\n",
        "In this cookbook, we will use Command-R to grade the completions. However, note that in practice, we typically use an ensemble of multiple LLM evaluators to reduce any bias."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "KGQQp2Yd13wr"
      },
      "outputs": [],
      "source": [
        "# LLM evaluation uses an LLM to grade the completion. We pass the original prompt, completion, and a\n",
        "# criteria to the LLM, and it returns a score between 1 and 5. We will use the following template as\n",
        "# the grading prompt.\n",
        "\n",
        "grading_prompt_template = \"\"\"You are an AI grader that given a prompt, a completion, and a criterion, grades the completion based on the prompt and criterion. Below is a prompt, a completion, and a criterion with which to grade the completion. You need to respond according to the criterion instructions.\n",
        "\n",
        "PROMPT:\n",
        "{prompt}\n",
        "\n",
        "COMPLETION:\n",
        "{completion}\n",
        "\n",
        "CRITERION:\n",
        "{criteria}\n",
        "\n",
        "Do not provide an explanation or respond with additional text. Only respond with the final score in the format SCORE: <final_score>\n",
        "\n",
        "SCORE:\"\"\"\n",
        "\n",
        "\n",
        "criteria_completeness = \"\"\"Please judge the above completion on whether or not all the completion fully addresses the task posed in the prompt. The completion should contain all the information that is asked for by the prompt while also upholding the constraints mentioned in the prompt. Please return a grade between 1 and 5, where 1 means the completion does not address the prompt at all, 2 means the completion mostly does not address the prompt, 3 is neutral, 4 means the completion mostly addresses the prompt, and 5 means the completion fully addresses the prompt.\"\"\"\n",
        "\n",
        "criteria_correctness = \"\"\"Please judge the above completion on whether or not all information was derived from the prompt and that no information is false. Return a grade between 1 and 5, where 1 means the answers are entirely untruthful relative to the prompt, 2 means the answers are somewhat untruthful relative to the prompt, 3 means neutral, 4 means the answers are somewhat truthful relative to the prompt, and 5 means the answers are entirely truthful relative to the prompt. Evaluate only the truthfulness of the answers, not whether or not they capture all the relevant information in the prompt.\"\"\"\n",
        "\n",
        "criteria_conciseness = \"\"\"Please judge the above completion on whether or not the completion contains any unnecessary information or wordiness that does not help answer the specific instruction given in the prompt. Return a grade between 1 and 5, where 1 means the completion contains many unnecessary details and wordiness that do not answer the specific instruction given in the prompt, 2 means the completion contains some unnecessary details or wordiness, 3 means neutral, 4 means the completion contains few unnecessary details or wordiness, and 5 means the completion contains only necessary details that answer the specific instruction given in the prompt.\"\"\"\n",
        "\n",
        "\n",
        "def score_llm(prompt: str, completion: str, criteria: str) -> int:\n",
        "    \"\"\"\n",
        "    Score a completion based on a prompt and a criterion using LLM Because we\n",
        "    grade all completions on a scale of 1-5, we will normalize the scores by 5 so that the final score\n",
        "    is between 0 and 1.\n",
        "    \"\"\"\n",
        "    grading_prompt = grading_prompt_template.format(\n",
        "        prompt=prompt, completion=completion, criteria=criteria\n",
        "    )\n",
        "    # Use Cohere to grade the completion\n",
        "    completion = co.chat(message=grading_prompt, model=co_model, temperature=0.2).text\n",
        "\n",
        "    ### Alternatively, use OpenAI to grade the completion (requires key)\n",
        "    # import openai\n",
        "    # completion = openai.OpenAI(api_key=\"INSERT OPENAI KEY HERE\").chat.completions.create(\n",
        "    #     model=\"gpt-4\",\n",
        "    #     messages=[{\"role\": \"user\", \"content\": grading_prompt}],\n",
        "    #     temperature=0.2,\n",
        "    # ).choices[0].message.content\n",
        "\n",
        "    # Extract the score from the completion\n",
        "    score = float(re.search(r\"[12345]\", completion).group()) / 5\n",
        "    return score"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "22dEEsXt13wr"
      },
      "source": [
        "In addition, we have two criteria that are graded programmatically:\n",
        "- Format: checks if the summary follows the format (e.g. bullets) that was requested in the prompt\n",
        "- Length: checks if the summary follows the length that was requested in the prompt."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "i1niSoI-13wr"
      },
      "outputs": [],
      "source": [
        "# Non-LLM checks\n",
        "\n",
        "def score_format(completion: str, format_type: str) -> int:\n",
        "    \"\"\"\n",
        "    Returns 1 if the completion is in the correct format, 0 otherwise.\n",
        "    \"\"\"\n",
        "    if format_type == \"paragraphs\":\n",
        "        return int(_is_only_paragraphs(completion))\n",
        "    elif format_type == \"bullets\":\n",
        "        return int(_is_only_bullets(completion))\n",
        "    return 0\n",
        "\n",
        "def score_length(\n",
        "    completion: str,\n",
        "    format_type: str,\n",
        "    min_val: int,\n",
        "    max_val: int,\n",
        "    number: Optional[int] = None\n",
        ") -> int:\n",
        "    \"\"\"\n",
        "    Returns 1 if the completion has the correct length for the given format, 0 otherwise. This\n",
        "    includes both word count and number of items (optional).\n",
        "    \"\"\"\n",
        "    # Split into items (each bullet for bullets or each paragraph for paragraphs)\n",
        "    if format_type == \"bullets\":\n",
        "        items = _extract_markdown_bullets(completion, include_bullet=False)\n",
        "    elif format_type == \"paragraphs\":\n",
        "        items = completion.split(\"\\n\")\n",
        "\n",
        "    # Strip whitespace and remove empty items\n",
        "    items = [item for item in items if item.strip() != \"\"]\n",
        "\n",
        "    # Check number of items if provided\n",
        "    if number is not None and len(items) != number:\n",
        "        return 0\n",
        "\n",
        "    # Check length of each item\n",
        "    for item in items:\n",
        "        num_words = item.strip().split()\n",
        "        if min_val is None and len(num_words) > max_val:\n",
        "            return 0\n",
        "        elif max_val is None and len(num_words) < min_val:\n",
        "            return 0\n",
        "        elif not min_val <= len(num_words) <= max_val:\n",
        "            return 0\n",
        "    return 1\n",
        "\n",
        "\n",
        "def _is_only_bullets(text: str) -> bool:\n",
        "    \"\"\"\n",
        "    Returns True if text is only markdown bullets.\n",
        "    \"\"\"\n",
        "    bullets = _extract_markdown_bullets(text, include_bullet=True)\n",
        "\n",
        "    for bullet in bullets:\n",
        "        text = text.replace(bullet, \"\")\n",
        "\n",
        "    return text.strip() == \"\"\n",
        "\n",
        "\n",
        "def _is_only_paragraphs(text: str) -> bool:\n",
        "    \"\"\"\n",
        "    Returns True if text is only paragraphs (no bullets).\n",
        "    \"\"\"\n",
        "    bullets = _extract_markdown_bullets(text, include_bullet=True)\n",
        "\n",
        "    return len(bullets) == 0\n",
        "\n",
        "\n",
        "def _extract_markdown_bullets(text: str, include_bullet: bool = False) -> List[str]:\n",
        "    \"\"\"\n",
        "    Extracts markdown bullets from text as a list. If include_bullet is True, the bullet will be\n",
        "    included in the output. The list of accepted bullets is: -, *, +, •, and any number followed by\n",
        "    a period.\n",
        "    \"\"\"\n",
        "    if include_bullet:\n",
        "        return re.findall(r\"^[ \\t]*(?:[-*+•]|[\\d]+\\.).*\\w+.*$\", text, flags=re.MULTILINE)\n",
        "    return re.findall(r\"^[ \\t]*(?:[-*+•]|[\\d]+\\.)(.*\\w+.*)$\", text, flags=re.MULTILINE)\n"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "kBpk-GTe13ws"
      },
      "source": [
        "<a id=\"run-evals\"></a>\n",
        "<a name=\"run-evals\"></a>\n",
        "# Run evaluations\n",
        "\n",
        "Now that we have our evaluation dataset and defined our evaluation functions, let's run evaluations!\n",
        "\n",
        "First, we generate completions to be graded. We will use Cohere's [Command-R](https://huggingface.co/CohereForAI/c4ai-command-r-v01) model, boasting a context length of 128K."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "HzbOQmWE13ws"
      },
      "outputs": [],
      "source": [
        "completions = []\n",
        "for prompt in data[\"prompt\"]:\n",
        "    completion = co.chat(message=prompt, model=\"command-r\", temperature=0.2).text\n",
        "    completions.append(completion)\n",
        "\n",
        "data[\"completion\"] = completions"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "JbVA8_Xr13ws",
        "outputId": "cebab3f2-4b02-4044-978e-3ae3ae2efa2d"
      },
      "outputs": [
        {
          "name": "stdout",
          "output_type": "stream",
          "text": [
            "PhD student D presented their method for detecting and counting acoustic events in recorded speech sessions, including speech overlaps and silence. The team sought clarification on D's definitions and counted events, discussing the challenges of distinguishing between different types of events.\n"
          ]
        }
      ],
      "source": [
        "# Let's look at the completion for the example prompt we showed earlier\n",
        "print(data[\"completion\"][0])"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6fWjuruK13ws"
      },
      "source": [
        "Let's grade the completions using our LLM and non-LLM checks."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "id": "6XesZHCN13ws"
      },
      "outputs": [],
      "source": [
        "# Score format\n",
        "data[\"format_score\"] = data.apply(\n",
        "    lambda x: score_format(x[\"completion\"], x[\"eval_metadata\"][\"format\"]), axis=1\n",
        ")\n",
        "\n",
        "# Score length\n",
        "data[\"length_score\"] = data.apply(\n",
        "    lambda x: score_length(\n",
        "        x[\"completion\"],\n",
        "        x[\"eval_metadata\"][\"format\"],\n",
        "        x[\"eval_metadata\"].get(\"min_length\"),\n",
        "        x[\"eval_metadata\"].get(\"max_length\"),\n",
        "    ),\n",
        "    axis=1,\n",
        ")\n",
        "\n",
        "# Score completeness\n",
        "data[\"completeness_score\"] = data.apply(\n",
        "    lambda x: score_llm(x[\"prompt\"], x[\"completion\"], criteria_completeness), axis=1\n",
        ")\n",
        "\n",
        "# Score correctness\n",
        "data[\"correctness_score\"] = data.apply(\n",
        "    lambda x: score_llm(x[\"prompt\"], x[\"completion\"], criteria_correctness), axis=1\n",
        ")\n",
        "\n",
        "# Score conciseness\n",
        "data[\"conciseness_score\"] = data.apply(\n",
        "    lambda x: score_llm(x[\"prompt\"], x[\"completion\"], criteria_conciseness), axis=1\n",
        ")"
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 657
        },
        "id": "4gJypAVI13ws",
        "outputId": "67aae9c3-02eb-4a1f-efcb-ba6c4781d18c"
      },
      "outputs": [
        {
          "data": {
            "application/vnd.google.colaboratory.intrinsic+json": {
              "summary": "{\n  \"name\": \"data\",\n  \"rows\": 6,\n  \"fields\": [\n    {\n      \"column\": \"instruction\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"Summarize the meeting based on the transcript. In paragraph form, output your response. Use at least 10 words and at most 50 words in total.\",\n          \"Summarize the meeting based on the transcript. Return the answer in the form of paragraphs. Make sure your answer is between 50 and 200 words long.\",\n          \"What are the follow-up items based on the meeting transcript? In bullets, output your response. Make sure to use exactly 2 bullets. Make sure each bullet is between 20 and 80 words long.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"eval_metadata\",\n      \"properties\": {\n        \"dtype\": \"object\",\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"objective\",\n      \"properties\": {\n        \"dtype\": \"category\",\n        \"num_unique_values\": 2,\n        \"samples\": [\n          \"action_items\",\n          \"general_summarization\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"transcript\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"PhD F: As opposed to the rest of us \\nPhD D: Well comment OK I I remind that me my first objective eh in the project is to to study difference parameters to to find a a good solution to detect eh the overlapping zone in eh speech recorded But eh tsk comment ehhh comment In that way comment I I I begin to to study and to analyze the ehn the recorded speech eh the different session to to find and to locate and to mark eh the the different overlapping zone And eh so eh I was eh I am transcribing the the first session and I I have found eh eh one thousand acoustic events eh besides the overlapping zones eh I I I mean the eh breaths eh aspiration eh eh talk eh eh clap eh comment I do not know what is the different names eh you use to to name the the pause n speech \\nGrad G: Oh I do not think we ve been doing it at that level of detail So \\nPhD D: Eh I I I do I do not need to to to mmm to m to label the the different acoustic but I prefer because eh I would like to to study if eh I I will find eh eh a good eh parameters eh to detect overlapping I would like to to to test these parameters eh with the another eh eh acoustic events to nnn to eh to find what is the ehm the false eh the false eh hypothesis eh nnn which eh are produced when we use the the ehm this eh parameter eh I mean pitch eh eh difference eh feature \\nPhD A: You know I think some of these that are the nonspeech overlapping events may be difficult even for humans to tell that there s two there I mean if it s a tapping sound you would not necessarily or you know something like that it would be it might be hard to know that it was two separate events \\nGrad G: Well You were not talking about just overlaps were you ? You were just talking about acoustic events \\nPhD D: I I I I t I t I talk eh about eh acoustic events in general but eh my my objective eh will be eh to study eh overlapping zone Eh ? comment n Eh in twelve minutes I found eh eh one thousand acoustic events \\nProfessor E: How many overlaps were there in it ? No no how many of them were the overlaps of speech though ? \\nPhD D: How many ? Eh almost eh three hundred eh in one session in five eh in forty five minutes Alm Three hundred overlapping zone With the overlapping zone overlapping speech speech what eh different duration \\nPostdoc B: Does this ? So if you had an overlap involving three people how many times was that counted ? \\nPhD D: three people two people Eh I would like to consider eh one people with difference noise eh in the background be \\nProfessor E: No no but I think what she s asking is pause if at some particular for some particular stretch you had three people talking instead of two did you call that one event ? \\nPhD D: Oh Oh I consider one event eh for th for that eh for all the zone This th I I I con I consider I consider eh an acoustic event the overlapping zone the period where three speaker or eh are talking together \\nGrad G: So let s say me and Jane are talking at the same time and then Liz starts talking also over all of us How many events would that be ? \\nPhD D: So I do not understand \\nGrad G: So two people are talking comment and then a third person starts talking Is there an event right here ? \\nPhD D: Eh no No no For me is the overlapping zone because because you you have s you have more one eh more one voice eh eh produced in a in in a moment \\nGrad G: So i if two or more people are talking \\nProfessor E: OK So I think We just wanted to understand how you are defining it So then in the region between since there there is some continuous region in between regions where there is only one person speaking And one contiguous region like that you are calling an event Is it Are you calling the beginning or the end of it the event or are you calling the entire length of it the event ? \\nPhD D: I consider the the nnn the nnn nnn eh the entirety eh eh all all the time there were the voice has overlapped This is the idea But eh I I do not distinguish between the the numbers of eh speaker I m not considering eh the the ehm eh the fact of eh eh for example what did you say ? Eh at first eh eh two talkers are eh speaking and eh eh a third person eh join to to that For me it s eh it s eh all overlap zone with eh several numbers of speakers is eh eh the same acoustic event Wi but without any mark between the zone of the overlapping zone with two speakers eh speaking together and the zone with the three speakers \\nPostdoc B: That would j just be one \\nPhD D: Eh with eh a beginning mark and the ending mark Because eh for me is the is the zone with eh some kind of eh distortion the spectral I do not mind By the moment by the moment \\nGrad G: Well but But you could imagine that three people talking has a different spectral characteristic than two \\nPhD D: I I do not but eh but eh I have to study comment What will happen in a general way \\nGrad G: So You had to start somewhere \\nPhD C: So there s a lot of overlap \\nPhD D: I I do not know what eh will will happen with the \\nGrad G: That s a lot of overlap \\nProfessor E: So again that s that s three three hundred in forty five minutes that are that are speakers just speakers \\nPostdoc B: But a a a th \\nProfessor E: So that s about eight per minute \\nPostdoc B: But a thousand events in twelve minutes that s \\nPhD C: But that can include taps \\nPostdoc B: Well but a thousand taps in eight minutes is a l in twelve minutes is a lot \\nPhD D: I I con I consider I consider acoustic events eh the silent too \\nGrad G: Silence starting or silence ending \\nPhD D: silent ground to bec to detect eh because I consider acoustic event all the things are not eh speech In ge in in in a general point of view \\nProfessor E: OK so how many of those thousand were silence ? \\nPhD F: Not speech not speech or too much speech \\nProfessor E: Right So how many of those thousand were silence silent sections ? \\nPhD D: silent I I I I do not I I have not the eh I I would like to to do a stylistic study\",\n          \"Lynne Neagle AM: Thank you very much And the next questions then are from Suzy Davies \\nSuzy Davies AM: Thank you Chair I just wanted to have a quick answer from probably the Minister I think about the primary legislation and the regulations that followed about which childrens rights impact assessments have been done Have any been done and can they be shared with the committee if they have ? Sorry Deputy Minister\\u2014my mistake \\nJulie Morgan AM: Well it is been a very difficult time as you appreciate in terms of having to make legislation very quickly and it has not been possible to do the impact assessments that we would normally do However I am very pleased to say that we are actually launching a survey of children We are going to be launching it next week And this is to try to get from children their views of what is happened what we have been doing and their views on the whole COVID19 situation So we are doing this in conjunction with the childrens commissioner and with Young Wales and with the Youth Parliament So this is an online survey that we hope will be going out to thousands of children and we will get their response in terms of what are the important issues that have arisen for them what they feel about what is happened during this period what they feel about the way that we have dealt with the schools the way that they have had to cope in not going school and being at home for so long And so we are trying to get feedback from young people So I am very pleased that we are doing that but in terms of an impact assessment it has been very difficult as I am sure you can imagine to be able to do those at these times I think that Albert wants to come in on that \\nSuzy Davies AM: Yes because I will pursue that in a sec \\nAlbert Heaney: Thank you Thank you Chair and I think Nicola indicated before me so apologies Nicola Just to say for the committee really importantly that we have not introduced any easements in relation to childrens services legislation I think that is really quite crucial So from a Welsh context the standards that are in place do remain so therefore there would not have been a necessity for us to do a childrens rights impact assessment in relation to the primary legislation I think that is particularly a strong point to us in Wales both in terms of safeguarding arrangements but also ensuring that childrens rights are protected at a crucial time \\nNicola Edwards: Thanks In terms of childcare and education we are obviously looking at the provisions under the coronavirus Act to allow us to maybe ease some of the statutory requirements and we are going to be undertaking a full suite of impact assessments on those Obviously the coronavirus Act itself was UK Government legislation and they ran their own impact assessments but in terms of how we implement it in the childcare and education space\\u2014and I think Albert was just saying the same thing\\u2014we definitely will be looking at those impacts in terms of going forward \\nSuzy Davies AM: Well just to come back on that then are you saying to me that as a result of the various coronavirus regulations that we have had no assessments for childrens needs have been postponed cancelled or done very quickly online rather than in person ? \\nJulie Morgan AM: Well I think as Albert said that there was no relaxation of regulation for childrens social care You know that is\\u2014there have not been any in Wales \\nSuzy Davies AM: No but that is what\\u2014 There is no relaxation but what is happening in practice ? We are down on staff across all our councils and in our third sectors\\u2014who is doing the childrens needs assessments particularly for young carers ? \\nJulie Morgan AM: Well I\\u2014 Albert can you answer that ? \\nAlbert Heaney: I think the first thing to say to the committee is that going back we took a very strong line at the beginning that we were not going to introduce easements in requirements to childrens social services Of course through the way that practitioners and social work practitioners have to operate they are having to operate through a different time So assessments are still taking place for child protection and safeguarding concerns assessments are still taking place and especially in relation to\\u2014as you mentioned\\u2014young carers to support their needs So arrangements\\u2014Inaudible But they are having to be slightly differently done\\u2014so some of the technology and keeping in contact and keeping those visits So we have used for example the St Davids Day fund to make sure that care leavers are well supported in terms of having contact and are accessible and able to engage as well So we are having to be a little bit more\\u2014and social services departments are having to be a little bit more\\u2014innovative in the use of technology in the way that they have engaged as well But personal visits are taking place and visits especially as the Minister mentioned earlier on\\u2014they actually individually assess each case to determine the frequency of visits to make sure that those contacts are maintained with children at a critical time \\nSuzy Davies AM: Thank you I do not want to take this much further but personal visits and social distancing could be slightly problematic I just want to finish with this one question if I may We have had recommendations from the Carers Trust or Carers Trust Wales Have they been accepted by Government and is it those that are driving the agenda of the task and finish group that you announced the other day Deputy Minister ? \\nJulie Morgan AM: Well those will certainly be considered by the task and finish group I have had a letter from the Carers Trust about those issues and we are setting up this group as you know and we will be looking at those issues in the group \\nSuzy Davies AM: Thank you Any steal on when that might report ? \\nJulie Morgan AM: I do not have that at the moment \\nLynne Neagle AM: Maybe we could have a note on that Deputy Minister Can I just say we are running short of time ? We did start late so if the Ministers are happy we will carry on until 210 pm\\u2014310 pm\\u2014if that is And the next questions are from Si\\u00e2n Gwenllian Hold on a sec Si\\u00e2n we have lost translation again Can we just see what can be done to get the translation back ? Sorry Si\\u00e2n Is there anyone who can help with the translation ? There you go Si\\u00e2n Thank you \\nSian Gwenllian AM: You will know Deputy Minister\\u2014because we have discussed this in private session\\u2014my major concerns with regard to the childcare sector and what kind of childcare sector we will have at the end of this crisis as families start to return to the workplace There are still some childcare providers who are falling between the cracks and are not receiving financial support Do you agree\\u2014are there people who are still not being supported and why is not the Welsh Government able to provide that support for everyone in the childcare sector ? \\nJulie Morgan AM: Thank you Si\\u00e2n for that question And I know that we have had a discussion about this before Basically we are aware that there are some sectors in the childcare sector that do fall through some of the loops We have guaranteed that we will pay the money for the childcare offer for three months So that is guaranteed to them and they are able to take advantage of the Governments job retainer scheme but that does mean that there is a problem as I think we have discussed before of the double funding issue and that is something that we have been trying to resolve and there have been discussions with the Treasury in Whitehall about ways forward on this I am going to ask Nicola to come in in a minute because she is much more up to date with the discussions about that but so far I do not think very much progress has been made on that But we are looking to see if there are any other ways that we can get help to the childcare sector and I am actually following this meeting with a meeting with the Deputy Minister for equality and chief whip who is responsible for the voluntary sector because obviously many of the groups that we are talking about would come under the voluntary sector because they have voluntary committees but they fall between many stools because they rent premises rather than own premises and they do not have high turnovers that would qualify them for some of these grants So perhaps I could ask Nicola to come in to expand on that\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"prompt\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"## meeting transcript\\nPhD F: As opposed to the rest of us \\nPhD D: Well comment OK I I remind that me my first objective eh in the project is to to study difference parameters to to find a a good solution to detect eh the overlapping zone in eh speech recorded But eh tsk comment ehhh comment In that way comment I I I begin to to study and to analyze the ehn the recorded speech eh the different session to to find and to locate and to mark eh the the different overlapping zone And eh so eh I was eh I am transcribing the the first session and I I have found eh eh one thousand acoustic events eh besides the overlapping zones eh I I I mean the eh breaths eh aspiration eh eh talk eh eh clap eh comment I do not know what is the different names eh you use to to name the the pause n speech \\nGrad G: Oh I do not think we ve been doing it at that level of detail So \\nPhD D: Eh I I I do I do not need to to to mmm to m to label the the different acoustic but I prefer because eh I would like to to study if eh I I will find eh eh a good eh parameters eh to detect overlapping I would like to to to test these parameters eh with the another eh eh acoustic events to nnn to eh to find what is the ehm the false eh the false eh hypothesis eh nnn which eh are produced when we use the the ehm this eh parameter eh I mean pitch eh eh difference eh feature \\nPhD A: You know I think some of these that are the nonspeech overlapping events may be difficult even for humans to tell that there s two there I mean if it s a tapping sound you would not necessarily or you know something like that it would be it might be hard to know that it was two separate events \\nGrad G: Well You were not talking about just overlaps were you ? You were just talking about acoustic events \\nPhD D: I I I I t I t I talk eh about eh acoustic events in general but eh my my objective eh will be eh to study eh overlapping zone Eh ? comment n Eh in twelve minutes I found eh eh one thousand acoustic events \\nProfessor E: How many overlaps were there in it ? No no how many of them were the overlaps of speech though ? \\nPhD D: How many ? Eh almost eh three hundred eh in one session in five eh in forty five minutes Alm Three hundred overlapping zone With the overlapping zone overlapping speech speech what eh different duration \\nPostdoc B: Does this ? So if you had an overlap involving three people how many times was that counted ? \\nPhD D: three people two people Eh I would like to consider eh one people with difference noise eh in the background be \\nProfessor E: No no but I think what she s asking is pause if at some particular for some particular stretch you had three people talking instead of two did you call that one event ? \\nPhD D: Oh Oh I consider one event eh for th for that eh for all the zone This th I I I con I consider I consider eh an acoustic event the overlapping zone the period where three speaker or eh are talking together \\nGrad G: So let s say me and Jane are talking at the same time and then Liz starts talking also over all of us How many events would that be ? \\nPhD D: So I do not understand \\nGrad G: So two people are talking comment and then a third person starts talking Is there an event right here ? \\nPhD D: Eh no No no For me is the overlapping zone because because you you have s you have more one eh more one voice eh eh produced in a in in a moment \\nGrad G: So i if two or more people are talking \\nProfessor E: OK So I think We just wanted to understand how you are defining it So then in the region between since there there is some continuous region in between regions where there is only one person speaking And one contiguous region like that you are calling an event Is it Are you calling the beginning or the end of it the event or are you calling the entire length of it the event ? \\nPhD D: I consider the the nnn the nnn nnn eh the entirety eh eh all all the time there were the voice has overlapped This is the idea But eh I I do not distinguish between the the numbers of eh speaker I m not considering eh the the ehm eh the fact of eh eh for example what did you say ? Eh at first eh eh two talkers are eh speaking and eh eh a third person eh join to to that For me it s eh it s eh all overlap zone with eh several numbers of speakers is eh eh the same acoustic event Wi but without any mark between the zone of the overlapping zone with two speakers eh speaking together and the zone with the three speakers \\nPostdoc B: That would j just be one \\nPhD D: Eh with eh a beginning mark and the ending mark Because eh for me is the is the zone with eh some kind of eh distortion the spectral I do not mind By the moment by the moment \\nGrad G: Well but But you could imagine that three people talking has a different spectral characteristic than two \\nPhD D: I I do not but eh but eh I have to study comment What will happen in a general way \\nGrad G: So You had to start somewhere \\nPhD C: So there s a lot of overlap \\nPhD D: I I do not know what eh will will happen with the \\nGrad G: That s a lot of overlap \\nProfessor E: So again that s that s three three hundred in forty five minutes that are that are speakers just speakers \\nPostdoc B: But a a a th \\nProfessor E: So that s about eight per minute \\nPostdoc B: But a thousand events in twelve minutes that s \\nPhD C: But that can include taps \\nPostdoc B: Well but a thousand taps in eight minutes is a l in twelve minutes is a lot \\nPhD D: I I con I consider I consider acoustic events eh the silent too \\nGrad G: Silence starting or silence ending \\nPhD D: silent ground to bec to detect eh because I consider acoustic event all the things are not eh speech In ge in in in a general point of view \\nProfessor E: OK so how many of those thousand were silence ? \\nPhD F: Not speech not speech or too much speech \\nProfessor E: Right So how many of those thousand were silence silent sections ? \\nPhD D: silent I I I I do not I I have not the eh I I would like to to do a stylistic study\\n\\n## instructions\\nSummarize the meeting based on the transcript. In paragraph form, output your response. Use at least 10 words and at most 50 words in total.\",\n          \"## meeting transcript\\nLynne Neagle AM: Thank you very much And the next questions then are from Suzy Davies \\nSuzy Davies AM: Thank you Chair I just wanted to have a quick answer from probably the Minister I think about the primary legislation and the regulations that followed about which childrens rights impact assessments have been done Have any been done and can they be shared with the committee if they have ? Sorry Deputy Minister\\u2014my mistake \\nJulie Morgan AM: Well it is been a very difficult time as you appreciate in terms of having to make legislation very quickly and it has not been possible to do the impact assessments that we would normally do However I am very pleased to say that we are actually launching a survey of children We are going to be launching it next week And this is to try to get from children their views of what is happened what we have been doing and their views on the whole COVID19 situation So we are doing this in conjunction with the childrens commissioner and with Young Wales and with the Youth Parliament So this is an online survey that we hope will be going out to thousands of children and we will get their response in terms of what are the important issues that have arisen for them what they feel about what is happened during this period what they feel about the way that we have dealt with the schools the way that they have had to cope in not going school and being at home for so long And so we are trying to get feedback from young people So I am very pleased that we are doing that but in terms of an impact assessment it has been very difficult as I am sure you can imagine to be able to do those at these times I think that Albert wants to come in on that \\nSuzy Davies AM: Yes because I will pursue that in a sec \\nAlbert Heaney: Thank you Thank you Chair and I think Nicola indicated before me so apologies Nicola Just to say for the committee really importantly that we have not introduced any easements in relation to childrens services legislation I think that is really quite crucial So from a Welsh context the standards that are in place do remain so therefore there would not have been a necessity for us to do a childrens rights impact assessment in relation to the primary legislation I think that is particularly a strong point to us in Wales both in terms of safeguarding arrangements but also ensuring that childrens rights are protected at a crucial time \\nNicola Edwards: Thanks In terms of childcare and education we are obviously looking at the provisions under the coronavirus Act to allow us to maybe ease some of the statutory requirements and we are going to be undertaking a full suite of impact assessments on those Obviously the coronavirus Act itself was UK Government legislation and they ran their own impact assessments but in terms of how we implement it in the childcare and education space\\u2014and I think Albert was just saying the same thing\\u2014we definitely will be looking at those impacts in terms of going forward \\nSuzy Davies AM: Well just to come back on that then are you saying to me that as a result of the various coronavirus regulations that we have had no assessments for childrens needs have been postponed cancelled or done very quickly online rather than in person ? \\nJulie Morgan AM: Well I think as Albert said that there was no relaxation of regulation for childrens social care You know that is\\u2014there have not been any in Wales \\nSuzy Davies AM: No but that is what\\u2014 There is no relaxation but what is happening in practice ? We are down on staff across all our councils and in our third sectors\\u2014who is doing the childrens needs assessments particularly for young carers ? \\nJulie Morgan AM: Well I\\u2014 Albert can you answer that ? \\nAlbert Heaney: I think the first thing to say to the committee is that going back we took a very strong line at the beginning that we were not going to introduce easements in requirements to childrens social services Of course through the way that practitioners and social work practitioners have to operate they are having to operate through a different time So assessments are still taking place for child protection and safeguarding concerns assessments are still taking place and especially in relation to\\u2014as you mentioned\\u2014young carers to support their needs So arrangements\\u2014Inaudible But they are having to be slightly differently done\\u2014so some of the technology and keeping in contact and keeping those visits So we have used for example the St Davids Day fund to make sure that care leavers are well supported in terms of having contact and are accessible and able to engage as well So we are having to be a little bit more\\u2014and social services departments are having to be a little bit more\\u2014innovative in the use of technology in the way that they have engaged as well But personal visits are taking place and visits especially as the Minister mentioned earlier on\\u2014they actually individually assess each case to determine the frequency of visits to make sure that those contacts are maintained with children at a critical time \\nSuzy Davies AM: Thank you I do not want to take this much further but personal visits and social distancing could be slightly problematic I just want to finish with this one question if I may We have had recommendations from the Carers Trust or Carers Trust Wales Have they been accepted by Government and is it those that are driving the agenda of the task and finish group that you announced the other day Deputy Minister ? \\nJulie Morgan AM: Well those will certainly be considered by the task and finish group I have had a letter from the Carers Trust about those issues and we are setting up this group as you know and we will be looking at those issues in the group \\nSuzy Davies AM: Thank you Any steal on when that might report ? \\nJulie Morgan AM: I do not have that at the moment \\nLynne Neagle AM: Maybe we could have a note on that Deputy Minister Can I just say we are running short of time ? We did start late so if the Ministers are happy we will carry on until 210 pm\\u2014310 pm\\u2014if that is And the next questions are from Si\\u00e2n Gwenllian Hold on a sec Si\\u00e2n we have lost translation again Can we just see what can be done to get the translation back ? Sorry Si\\u00e2n Is there anyone who can help with the translation ? There you go Si\\u00e2n Thank you \\nSian Gwenllian AM: You will know Deputy Minister\\u2014because we have discussed this in private session\\u2014my major concerns with regard to the childcare sector and what kind of childcare sector we will have at the end of this crisis as families start to return to the workplace There are still some childcare providers who are falling between the cracks and are not receiving financial support Do you agree\\u2014are there people who are still not being supported and why is not the Welsh Government able to provide that support for everyone in the childcare sector ? \\nJulie Morgan AM: Thank you Si\\u00e2n for that question And I know that we have had a discussion about this before Basically we are aware that there are some sectors in the childcare sector that do fall through some of the loops We have guaranteed that we will pay the money for the childcare offer for three months So that is guaranteed to them and they are able to take advantage of the Governments job retainer scheme but that does mean that there is a problem as I think we have discussed before of the double funding issue and that is something that we have been trying to resolve and there have been discussions with the Treasury in Whitehall about ways forward on this I am going to ask Nicola to come in in a minute because she is much more up to date with the discussions about that but so far I do not think very much progress has been made on that But we are looking to see if there are any other ways that we can get help to the childcare sector and I am actually following this meeting with a meeting with the Deputy Minister for equality and chief whip who is responsible for the voluntary sector because obviously many of the groups that we are talking about would come under the voluntary sector because they have voluntary committees but they fall between many stools because they rent premises rather than own premises and they do not have high turnovers that would qualify them for some of these grants So perhaps I could ask Nicola to come in to expand on that\\n\\n## instructions\\nSummarize the meeting based on the transcript. Return the answer in the form of paragraphs. Make sure your answer is between 50 and 200 words long.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"transcript_token_len\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 568,\n        \"min\": 1100,\n        \"max\": 2618,\n        \"num_unique_values\": 6,\n        \"samples\": [\n          1378,\n          1649\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"completion\",\n      \"properties\": {\n        \"dtype\": \"string\",\n        \"num_unique_values\": 6,\n        \"samples\": [\n          \"PhD student D presented their method for detecting and counting acoustic events in recorded speech sessions, including speech overlaps and silence. The team sought clarification on D's definitions and counted events, discussing the challenges of distinguishing between different types of events.\",\n          \"The meeting between Welsh politicians and officials centered around the impact of the coronavirus legislation on children's rights and the childcare sector in Wales. Julie Morgan, the AM, explained that a survey would be launched to gather the views of children on the pandemic situation, as it had been challenging to conduct children's rights impact assessments during these fast-paced times. \\n\\nAlbert Heaney emphasized that there had been no relaxation of children's social care regulations in Wales, but that the methods of assessment had to be adapted due to the pandemic. In response to Sian Gwenllian's concerns, Julie Morgan acknowledged that some childcare providers were struggling financially due to the pandemic. She assured that the government was trying to support them, but the complexity of the issue had hindered progress, especially regarding the double funding problem.\"\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"format_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 1,\n        \"max\": 1,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"length_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0,\n        \"min\": 0,\n        \"max\": 1,\n        \"num_unique_values\": 2,\n        \"samples\": [\n          0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"completeness_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.2161883888976234e-16,\n        \"min\": 0.8,\n        \"max\": 0.8,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.8\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"correctness_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 0.0,\n        \"min\": 1.0,\n        \"max\": 1.0,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          1.0\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    },\n    {\n      \"column\": \"conciseness_score\",\n      \"properties\": {\n        \"dtype\": \"number\",\n        \"std\": 1.2161883888976234e-16,\n        \"min\": 0.8,\n        \"max\": 0.8,\n        \"num_unique_values\": 1,\n        \"samples\": [\n          0.8\n        ],\n        \"semantic_type\": \"\",\n        \"description\": \"\"\n      }\n    }\n  ]\n}",
              "type": "dataframe",
              "variable_name": "data"
            },
            "text/html": [
              "\n",
              "  <div id=\"df-0bf386c2-7304-4a6e-a3ba-1fe955197e82\" class=\"colab-df-container\">\n",
              "    <div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>instruction</th>\n",
              "      <th>eval_metadata</th>\n",
              "      <th>objective</th>\n",
              "      <th>transcript</th>\n",
              "      <th>prompt</th>\n",
              "      <th>transcript_token_len</th>\n",
              "      <th>completion</th>\n",
              "      <th>format_score</th>\n",
              "      <th>length_score</th>\n",
              "      <th>completeness_score</th>\n",
              "      <th>correctness_score</th>\n",
              "      <th>conciseness_score</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Summarize the meeting based on the transcript....</td>\n",
              "      <td>{'format': 'paragraphs', 'min_length': 10, 'ma...</td>\n",
              "      <td>general_summarization</td>\n",
              "      <td>PhD F: As opposed to the rest of us \\nPhD D: W...</td>\n",
              "      <td>## meeting transcript\\nPhD F: As opposed to th...</td>\n",
              "      <td>1378</td>\n",
              "      <td>PhD student D presented their method for detec...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.8</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Summarize the meeting based on the transcript....</td>\n",
              "      <td>{'format': 'paragraphs', 'min_length': 50, 'ma...</td>\n",
              "      <td>general_summarization</td>\n",
              "      <td>Lynne Neagle AM: Thank you very much And the n...</td>\n",
              "      <td>## meeting transcript\\nLynne Neagle AM: Thank ...</td>\n",
              "      <td>1649</td>\n",
              "      <td>The meeting between Welsh politicians and offi...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.8</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Summarize the meeting based on the transcript....</td>\n",
              "      <td>{'format': 'bullets', 'number': 3, 'min_length...</td>\n",
              "      <td>general_summarization</td>\n",
              "      <td>Industrial Designer: Yep So we are to mainly d...</td>\n",
              "      <td>## meeting transcript\\nIndustrial Designer: Ye...</td>\n",
              "      <td>1100</td>\n",
              "      <td>- The team discusses the design of a remote co...</td>\n",
              "      <td>1</td>\n",
              "      <td>0</td>\n",
              "      <td>0.8</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Summarize the meeting based on the transcript....</td>\n",
              "      <td>{'format': 'bullets', 'number': 2, 'min_length...</td>\n",
              "      <td>general_summarization</td>\n",
              "      <td>Industrial Designer: Mm I think one of the ver...</td>\n",
              "      <td>## meeting transcript\\nIndustrial Designer: Mm...</td>\n",
              "      <td>2618</td>\n",
              "      <td>- The team discusses the target demographic fo...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.8</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>What are the follow-up items based on the meet...</td>\n",
              "      <td>{'format': 'bullets', 'number': 3, 'min_length...</td>\n",
              "      <td>action_items</td>\n",
              "      <td>Marketing: so a lot of people have to be able ...</td>\n",
              "      <td>## meeting transcript\\nMarketing: so a lot of ...</td>\n",
              "      <td>2286</td>\n",
              "      <td>- Investigate remote's compatibility with vari...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.8</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.8</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>5</th>\n",
              "      <td>What are the follow-up items based on the meet...</td>\n",
              "      <td>{'format': 'bullets', 'number': 2, 'min_length...</td>\n",
              "      <td>action_items</td>\n",
              "      <td>Project Manager: Alright So finance And we wil...</td>\n",
              "      <td>## meeting transcript\\nProject Manager: Alrigh...</td>\n",
              "      <td>1965</td>\n",
              "      <td>- The project manager will send the updated de...</td>\n",
              "      <td>1</td>\n",
              "      <td>1</td>\n",
              "      <td>0.8</td>\n",
              "      <td>1.0</td>\n",
              "      <td>0.8</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>\n",
              "    <div class=\"colab-df-buttons\">\n",
              "\n",
              "  <div class=\"colab-df-container\">\n",
              "    <button class=\"colab-df-convert\" onclick=\"convertToInteractive('df-0bf386c2-7304-4a6e-a3ba-1fe955197e82')\"\n",
              "            title=\"Convert this dataframe to an interactive table.\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "  <svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\" viewBox=\"0 -960 960 960\">\n",
              "    <path d=\"M120-120v-720h720v720H120Zm60-500h600v-160H180v160Zm220 220h160v-160H400v160Zm0 220h160v-160H400v160ZM180-400h160v-160H180v160Zm440 0h160v-160H620v160ZM180-180h160v-160H180v160Zm440 0h160v-160H620v160Z\"/>\n",
              "  </svg>\n",
              "    </button>\n",
              "\n",
              "  <style>\n",
              "    .colab-df-container {\n",
              "      display:flex;\n",
              "      gap: 12px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert {\n",
              "      background-color: #E8F0FE;\n",
              "      border: none;\n",
              "      border-radius: 50%;\n",
              "      cursor: pointer;\n",
              "      display: none;\n",
              "      fill: #1967D2;\n",
              "      height: 32px;\n",
              "      padding: 0 0 0 0;\n",
              "      width: 32px;\n",
              "    }\n",
              "\n",
              "    .colab-df-convert:hover {\n",
              "      background-color: #E2EBFA;\n",
              "      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "      fill: #174EA6;\n",
              "    }\n",
              "\n",
              "    .colab-df-buttons div {\n",
              "      margin-bottom: 4px;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert {\n",
              "      background-color: #3B4455;\n",
              "      fill: #D2E3FC;\n",
              "    }\n",
              "\n",
              "    [theme=dark] .colab-df-convert:hover {\n",
              "      background-color: #434B5C;\n",
              "      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);\n",
              "      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));\n",
              "      fill: #FFFFFF;\n",
              "    }\n",
              "  </style>\n",
              "\n",
              "    <script>\n",
              "      const buttonEl =\n",
              "        document.querySelector('#df-0bf386c2-7304-4a6e-a3ba-1fe955197e82 button.colab-df-convert');\n",
              "      buttonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "\n",
              "      async function convertToInteractive(key) {\n",
              "        const element = document.querySelector('#df-0bf386c2-7304-4a6e-a3ba-1fe955197e82');\n",
              "        const dataTable =\n",
              "          await google.colab.kernel.invokeFunction('convertToInteractive',\n",
              "                                                    [key], {});\n",
              "        if (!dataTable) return;\n",
              "\n",
              "        const docLinkHtml = 'Like what you see? Visit the ' +\n",
              "          '<a target=\"_blank\" href=https://colab.research.google.com/notebooks/data_table.ipynb>data table notebook</a>'\n",
              "          + ' to learn more about interactive tables.';\n",
              "        element.innerHTML = '';\n",
              "        dataTable['output_type'] = 'display_data';\n",
              "        await google.colab.output.renderOutput(dataTable, element);\n",
              "        const docLink = document.createElement('div');\n",
              "        docLink.innerHTML = docLinkHtml;\n",
              "        element.appendChild(docLink);\n",
              "      }\n",
              "    </script>\n",
              "  </div>\n",
              "\n",
              "\n",
              "<div id=\"df-c1880cbe-ec93-49b7-939e-947e33f06133\">\n",
              "  <button class=\"colab-df-quickchart\" onclick=\"quickchart('df-c1880cbe-ec93-49b7-939e-947e33f06133')\"\n",
              "            title=\"Suggest charts\"\n",
              "            style=\"display:none;\">\n",
              "\n",
              "<svg xmlns=\"http://www.w3.org/2000/svg\" height=\"24px\"viewBox=\"0 0 24 24\"\n",
              "     width=\"24px\">\n",
              "    <g>\n",
              "        <path d=\"M19 3H5c-1.1 0-2 .9-2 2v14c0 1.1.9 2 2 2h14c1.1 0 2-.9 2-2V5c0-1.1-.9-2-2-2zM9 17H7v-7h2v7zm4 0h-2V7h2v10zm4 0h-2v-4h2v4z\"/>\n",
              "    </g>\n",
              "</svg>\n",
              "  </button>\n",
              "\n",
              "<style>\n",
              "  .colab-df-quickchart {\n",
              "      --bg-color: #E8F0FE;\n",
              "      --fill-color: #1967D2;\n",
              "      --hover-bg-color: #E2EBFA;\n",
              "      --hover-fill-color: #174EA6;\n",
              "      --disabled-fill-color: #AAA;\n",
              "      --disabled-bg-color: #DDD;\n",
              "  }\n",
              "\n",
              "  [theme=dark] .colab-df-quickchart {\n",
              "      --bg-color: #3B4455;\n",
              "      --fill-color: #D2E3FC;\n",
              "      --hover-bg-color: #434B5C;\n",
              "      --hover-fill-color: #FFFFFF;\n",
              "      --disabled-bg-color: #3B4455;\n",
              "      --disabled-fill-color: #666;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart {\n",
              "    background-color: var(--bg-color);\n",
              "    border: none;\n",
              "    border-radius: 50%;\n",
              "    cursor: pointer;\n",
              "    display: none;\n",
              "    fill: var(--fill-color);\n",
              "    height: 32px;\n",
              "    padding: 0;\n",
              "    width: 32px;\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart:hover {\n",
              "    background-color: var(--hover-bg-color);\n",
              "    box-shadow: 0 1px 2px rgba(60, 64, 67, 0.3), 0 1px 3px 1px rgba(60, 64, 67, 0.15);\n",
              "    fill: var(--button-hover-fill-color);\n",
              "  }\n",
              "\n",
              "  .colab-df-quickchart-complete:disabled,\n",
              "  .colab-df-quickchart-complete:disabled:hover {\n",
              "    background-color: var(--disabled-bg-color);\n",
              "    fill: var(--disabled-fill-color);\n",
              "    box-shadow: none;\n",
              "  }\n",
              "\n",
              "  .colab-df-spinner {\n",
              "    border: 2px solid var(--fill-color);\n",
              "    border-color: transparent;\n",
              "    border-bottom-color: var(--fill-color);\n",
              "    animation:\n",
              "      spin 1s steps(1) infinite;\n",
              "  }\n",
              "\n",
              "  @keyframes spin {\n",
              "    0% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "      border-left-color: var(--fill-color);\n",
              "    }\n",
              "    20% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    30% {\n",
              "      border-color: transparent;\n",
              "      border-left-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    40% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-top-color: var(--fill-color);\n",
              "    }\n",
              "    60% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "    }\n",
              "    80% {\n",
              "      border-color: transparent;\n",
              "      border-right-color: var(--fill-color);\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "    90% {\n",
              "      border-color: transparent;\n",
              "      border-bottom-color: var(--fill-color);\n",
              "    }\n",
              "  }\n",
              "</style>\n",
              "\n",
              "  <script>\n",
              "    async function quickchart(key) {\n",
              "      const quickchartButtonEl =\n",
              "        document.querySelector('#' + key + ' button');\n",
              "      quickchartButtonEl.disabled = true;  // To prevent multiple clicks.\n",
              "      quickchartButtonEl.classList.add('colab-df-spinner');\n",
              "      try {\n",
              "        const charts = await google.colab.kernel.invokeFunction(\n",
              "            'suggestCharts', [key], {});\n",
              "      } catch (error) {\n",
              "        console.error('Error during call to suggestCharts:', error);\n",
              "      }\n",
              "      quickchartButtonEl.classList.remove('colab-df-spinner');\n",
              "      quickchartButtonEl.classList.add('colab-df-quickchart-complete');\n",
              "    }\n",
              "    (() => {\n",
              "      let quickchartButtonEl =\n",
              "        document.querySelector('#df-c1880cbe-ec93-49b7-939e-947e33f06133 button');\n",
              "      quickchartButtonEl.style.display =\n",
              "        google.colab.kernel.accessAllowed ? 'block' : 'none';\n",
              "    })();\n",
              "  </script>\n",
              "</div>\n",
              "    </div>\n",
              "  </div>\n"
            ],
            "text/plain": [
              "                                         instruction  \\\n",
              "0  Summarize the meeting based on the transcript....   \n",
              "1  Summarize the meeting based on the transcript....   \n",
              "2  Summarize the meeting based on the transcript....   \n",
              "3  Summarize the meeting based on the transcript....   \n",
              "4  What are the follow-up items based on the meet...   \n",
              "5  What are the follow-up items based on the meet...   \n",
              "\n",
              "                                       eval_metadata              objective  \\\n",
              "0  {'format': 'paragraphs', 'min_length': 10, 'ma...  general_summarization   \n",
              "1  {'format': 'paragraphs', 'min_length': 50, 'ma...  general_summarization   \n",
              "2  {'format': 'bullets', 'number': 3, 'min_length...  general_summarization   \n",
              "3  {'format': 'bullets', 'number': 2, 'min_length...  general_summarization   \n",
              "4  {'format': 'bullets', 'number': 3, 'min_length...           action_items   \n",
              "5  {'format': 'bullets', 'number': 2, 'min_length...           action_items   \n",
              "\n",
              "                                          transcript  \\\n",
              "0  PhD F: As opposed to the rest of us \\nPhD D: W...   \n",
              "1  Lynne Neagle AM: Thank you very much And the n...   \n",
              "2  Industrial Designer: Yep So we are to mainly d...   \n",
              "3  Industrial Designer: Mm I think one of the ver...   \n",
              "4  Marketing: so a lot of people have to be able ...   \n",
              "5  Project Manager: Alright So finance And we wil...   \n",
              "\n",
              "                                              prompt  transcript_token_len  \\\n",
              "0  ## meeting transcript\\nPhD F: As opposed to th...                  1378   \n",
              "1  ## meeting transcript\\nLynne Neagle AM: Thank ...                  1649   \n",
              "2  ## meeting transcript\\nIndustrial Designer: Ye...                  1100   \n",
              "3  ## meeting transcript\\nIndustrial Designer: Mm...                  2618   \n",
              "4  ## meeting transcript\\nMarketing: so a lot of ...                  2286   \n",
              "5  ## meeting transcript\\nProject Manager: Alrigh...                  1965   \n",
              "\n",
              "                                          completion  format_score  \\\n",
              "0  PhD student D presented their method for detec...             1   \n",
              "1  The meeting between Welsh politicians and offi...             1   \n",
              "2  - The team discusses the design of a remote co...             1   \n",
              "3  - The team discusses the target demographic fo...             1   \n",
              "4  - Investigate remote's compatibility with vari...             1   \n",
              "5  - The project manager will send the updated de...             1   \n",
              "\n",
              "   length_score  completeness_score  correctness_score  conciseness_score  \n",
              "0             1                 0.8                1.0                0.8  \n",
              "1             1                 0.8                1.0                0.8  \n",
              "2             0                 0.8                1.0                0.8  \n",
              "3             1                 0.8                1.0                0.8  \n",
              "4             1                 0.8                1.0                0.8  \n",
              "5             1                 0.8                1.0                0.8  "
            ]
          },
          "execution_count": 36,
          "metadata": {},
          "output_type": "execute_result"
        }
      ],
      "source": [
        "# Let's examine the final evaluation results\n",
        "data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "9bOUjcce13ws"
      },
      "source": [
        "Finally, let's plot the average scores per critiera."
      ]
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 598
        },
        "id": "J_MeQOWX13ws",
        "outputId": "8cedc547-6311-4700-c873-5dbbcbc6f52d"
      },
      "outputs": [
        {
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAjcAAAJFCAYAAAAhwtZkAAAAOXRFWHRTb2Z0d2FyZQBNYXRwbG90bGliIHZlcnNpb24zLjcuMSwgaHR0cHM6Ly9tYXRwbG90bGliLm9yZy/bCgiHAAAACXBIWXMAAA9hAAAPYQGoP6dpAABYOUlEQVR4nO3deVhUdeP+8XtA2UTcQTMecXtSEJckzf0xd00fzcqlxy2zTFxpUSu10kQtTUvT3LM092x1C3czcUMtzZXETFzDBQoEzu8Pf863CTQ1mcOceb+ui+uKz5wZbjgx3pzzOedjMwzDEAAAgEV4mB0AAADgXqLcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcABb3wQcfyGazqWbNmmZHyXXS0tI0adIkVatWTQEBASpYsKDCwsL07LPP6qeffjI7HoC7lMfsAABy1vz58xUSEqLY2FgdPXpU5cqVMztSrtG+fXutXLlSnTp1Uq9evXTt2jX99NNP+uqrr1S7dm1VqFDB7IgA7oKNhTMB64qPj1eZMmW0fPlyPffcc4qMjNSIESOcmiEzM1NpaWny8fFx6tf9Ozt27FCNGjX01ltv6ZVXXnF4LCMjQ0lJSSpSpIhTsvzxxx/y8vKShwcH04F7gd8kwMLmz5+vQoUKqVWrVnr88cc1f/58+2PXrl1T4cKF1aNHjyzPu3z5snx8fPTiiy/ax1JTUzVixAiVK1dO3t7eCg4O1ssvv6zU1FSH59psNvXt21fz589XWFiYvL29tWrVKknSO++8o9q1a6tIkSLy9fVV9erVtXTp0ixf//fff1f//v1VtGhR5c+fX23atNGpU6dks9n0+uuvO2x76tQpPf300woKCpK3t7fCwsI0e/bsv/3ZHDt2TJJUp06dLI95enpmKTanTp1Sz549dd9998nb21ulS5fW888/r7S0NPs2x48f1xNPPKHChQvLz89PDz/8sL7++muH19mwYYNsNpsWLlyo1157TSVLlpSfn58uX74sSdq+fbuaN2+uAgUKyM/PTw0aNNDWrVsdXuPKlSsaOHCgQkJC5O3trcDAQDVp0kS7d+/+2+8bcAeclgIsbP78+Xrsscfk5eWlTp06aerUqdqxY4ceeugh5c2bV+3atdPy5cv14YcfysvLy/68FStWKDU1VR07dpR0/ehLmzZttGXLFj377LOqWLGi9u/fr3fffVeHDx/WihUrHL7uunXrtHjxYvXt21dFixZVSEiIJGnSpElq06aNnnrqKaWlpWnhwoV64okn9NVXX6lVq1b253fv3l2LFy9Wly5d9PDDD2vjxo0Oj99w5swZPfzww/ZCVaxYMa1cuVI9e/bU5cuXNXDgwJv+bEqVKmX/GdWpU0d58tz87fDXX39VjRo1lJSUpGeffVYVKlTQqVOntHTpUqWkpMjLy0tnzpxR7dq1lZKSov79+6tIkSL66KOP1KZNGy1dulTt2rVzeM2RI0fKy8tLL774olJTU+Xl5aV169apRYsWql69ukaMGCEPDw/NmTNHjzzyiDZv3qwaNWpIknr37q2lS5eqb9++Cg0N1YULF7RlyxYdPHhQDz744E2/D8BtGAAsaefOnYYkY+3atYZhGEZmZqZx//33GwMGDLBvs3r1akOS8eWXXzo8t2XLlkaZMmXsn3/88ceGh4eHsXnzZoftpk2bZkgytm7dah+TZHh4eBg//vhjlkwpKSkOn6elpRmVKlUyHnnkEfvYrl27DEnGwIEDHbbt3r27IckYMWKEfaxnz55GiRIljPPnzzts27FjR6NAgQJZvt6fZWZmGg0aNDAkGUFBQUanTp2MKVOmGCdOnMiybdeuXQ0PDw9jx44d2b6OYRjGwIEDDUkOP6MrV64YpUuXNkJCQoyMjAzDMAxj/fr1hiSjTJkyDvkyMzON8uXLG82aNbO/5o2fWenSpY0mTZrYxwoUKGBERkbe9HsD3B2npQCLmj9/voKCgtSwYUNJ108XdejQQQsXLlRGRoYk6ZFHHlHRokW1aNEi+/N+++03rV27Vh06dLCPLVmyRBUrVlSFChV0/vx5+8cjjzwiSVq/fr3D127QoIFCQ0OzZPL19XX4OpcuXVK9evUcTqfcOIXVp08fh+f269fP4XPDMLRs2TK1bt1ahmE45GrWrJkuXbp0y9M0NptNq1ev1qhRo1SoUCF9+umnioyMVKlSpdShQwclJSVJun7UasWKFWrdurUiIiKyfR1J+uabb1SjRg3VrVvX/pi/v7+effZZ/fzzzzpw4IDD87p16+bw84iLi9ORI0fUuXNnXbhwwf69JCcnq1GjRtq0aZMyMzMlSQULFtT27dv166+/3vT7A9wZp6UAC8rIyNDChQvVsGFDxcfH28dr1qyp8ePHKyYmRk2bNlWePHnUvn17LViwQKmpqfL29tby5ct17do1h3Jz5MgRHTx4UMWKFcv26509e9bh89KlS2e73VdffaVRo0YpLi7OYa7OjYIgSSdOnJCHh0eW1/jrVV7nzp1TUlKSpk+frunTp99Wrr/y9vbWq6++qldffVWnT5/Wxo0bNWnSJC1evFh58+bVJ598onPnzuny5cuqVKnSLV/rxIkT2V5uX7FiRfvjf36Nv35/R44ckXS99NzMpUuXVKhQIY0bN07dunVTcHCwqlevrpYtW6pr164qU6bMLTMC7oJyA1jQunXrdPr0aS1cuFALFy7M8vj8+fPVtGlTSVLHjh314YcfauXKlWrbtq0WL16sChUqqEqVKvbtMzMzFR4ergkTJmT79YKDgx0+//MRiRs2b96sNm3aqH79+vrggw9UokQJ5c2bV3PmzNGCBQvu+Hu8cRTjf//7300LQeXKlW/79UqUKKGOHTuqffv2CgsL0+LFizV37tw7znW7/vozuvH9vP3226patWq2z/H395ckPfnkk6pXr54+++wzrVmzRm+//bbGjh2r5cuXq0WLFjmWGXAVlBvAgubPn6/AwEBNmTIly2PLly/XZ599pmnTpsnX11f169dXiRIltGjRItWtW1fr1q3Tq6++6vCcsmXLau/evWrUqJHDUZY7sWzZMvn4+Gj16tXy9va2j8+ZM8dhu1KlSikzM1Px8fEqX768ffzo0aMO2xUrVkz58+dXRkaGGjdufFeZspM3b15VrlxZR44c0fnz5xUYGKiAgAD98MMPt3xeqVKldOjQoSzjN24GeGMC882ULVtWkhQQEHBb30+JEiXUp08f9enTR2fPntWDDz6ot956i3IDiEvBAcv5/ffftXz5cj366KN6/PHHs3z07dtXV65c0RdffCFJ8vDw0OOPP64vv/xSH3/8sdLT0x1OSUnXjxScOnVKM2bMyPbrJScn/20uT09P2Ww2+3wfSfr555+zXGnVrFkzSdfvrPxn77//fpbXa9++vZYtW5Zt8Th37twt8xw5ckQJCQlZxpOSkrRt2zYVKlRIxYoVk4eHh9q2basvv/xSO3fuzLK98f9vFdayZUvFxsZq27Zt9seSk5M1ffp0hYSEZDsH6c+qV6+usmXL6p133tHVq1dv+v1kZGTo0qVLDo8FBgbqvvvuy3JZPuCuOHIDWMwXX3yhK1euqE2bNtk+/vDDD6tYsWKaP3++vcR06NBB77//vkaMGKHw8HD7PJEbunTposWLF6t3795av3696tSpo4yMDP30009avHixVq9ene1k2z9r1aqVJkyYoObNm6tz5846e/aspkyZonLlymnfvn327apXr6727dtr4sSJunDhgv1S8MOHD0tynJ8zZswYrV+/XjVr1lSvXr0UGhqqixcvavfu3fr222918eLFm+bZu3evOnfurBYtWqhevXoqXLiwTp06pY8++ki//vqrJk6cKE9PT0nS6NGjtWbNGjVo0MB+Kfzp06e1ZMkSbdmyRQULFtSQIUP06aefqkWLFurfv78KFy6sjz76SPHx8Vq2bNnf3qDPw8NDM2fOVIsWLRQWFqYePXqoZMmSOnXqlNavX6+AgAB9+eWXunLliu6//349/vjjqlKlivz9/fXtt99qx44dGj9+/C2/BuA2TL5aC8A91rp1a8PHx8dITk6+6Tbdu3c38ubNa7+EOjMz0wgODjYkGaNGjcr2OWlpacbYsWONsLAww9vb2yhUqJBRvXp144033jAuXbpk307STS9TnjVrllG+fHnD29vbqFChgjFnzhxjxIgRxl/fipKTk43IyEijcOHChr+/v9G2bVvj0KFDhiRjzJgxDtueOXPGiIyMNIKDg428efMaxYsXNxo1amRMnz79lj+nM2fOGGPGjDEaNGhglChRwsiTJ49RqFAh45FHHjGWLl2aZfsTJ04YXbt2NYoVK2Z4e3sbZcqUMSIjI43U1FT7NseOHTMef/xxo2DBgoaPj49Ro0YN46uvvnJ4nRuXgi9ZsiTbXHv27DEee+wxo0iRIoa3t7dRqlQp48knnzRiYmIMwzCM1NRU46WXXjKqVKli5M+f38iXL59RpUoV44MPPrjl9wu4E5ZfAOAS4uLiVK1aNX3yySd66qmnzI4DIBdjzg2AXOf333/PMjZx4kR5eHiofv36JiQC4EqYcwMg1xk3bpx27dqlhg0bKk+ePFq5cqVWrlypZ599Nstl5wDwV5yWApDrrF27Vm+88YYOHDigq1ev6l//+pe6dOmiV1999ZZrQAGARLkBAAAWw5wbAABgKW53fDczM1O//vqr8ufPf9d3WgUAAM5lGIauXLmi++6772/vG+V25ebXX39lQiIAAC7q5MmTuv/++2+5jduVm/z580u6/sMJCAgwOQ0AALgdly9fVnBwsP3f8Vtxu3Jz41RUQEAA5QYAABdzO1NKmFAMAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAshXIDAAAsxdRys2nTJrVu3Vr33XefbDabVqxY8bfP2bBhgx588EF5e3urXLlymjt3bo7nBAAArsPUcpOcnKwqVapoypQpt7V9fHy8WrVqpYYNGyouLk4DBw7UM888o9WrV+dwUgAA4CpMXRW8RYsWatGixW1vP23aNJUuXVrjx4+XJFWsWFFbtmzRu+++q2bNmuVUTAAA4EJcas7Ntm3b1LhxY4exZs2aadu2bTd9Tmpqqi5fvuzwAQAArMvUIzd3KjExUUFBQQ5jQUFBunz5sn7//Xf5+vpmeU50dLTeeOMNZ0W0CxnytdO/Zk74eUwrsyMAyCFWeJ/iPQrZcakjN3dj6NChunTpkv3j5MmTZkcCAAA5yKWO3BQvXlxnzpxxGDtz5owCAgKyPWojSd7e3vL29nZGPAAAkAu41JGbWrVqKSYmxmFs7dq1qlWrlkmJAABAbmNqubl69ari4uIUFxcn6fql3nFxcUpISJB0/ZRS165d7dv37t1bx48f18svv6yffvpJH3zwgRYvXqxBgwaZER8AAORCppabnTt3qlq1aqpWrZokKSoqStWqVdPw4cMlSadPn7YXHUkqXbq0vv76a61du1ZVqlTR+PHjNXPmTC4DBwAAdqbOufnPf/4jwzBu+nh2dx/+z3/+oz179uRgKgAA4Mpcas4NAADA36HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS6HcAAAAS8ljdgAgp4UM+drsCPfEz2NamR0BAFwCR24AAIClUG4AAIClUG4AAIClUG4AAIClUG4AAIClUG4AAIClUG4AAIClmF5upkyZopCQEPn4+KhmzZqKjY295fYTJ07UAw88IF9fXwUHB2vQoEH6448/nJQWAADkdqaWm0WLFikqKkojRozQ7t27VaVKFTVr1kxnz57NdvsFCxZoyJAhGjFihA4ePKhZs2Zp0aJFeuWVV5ycHAAA5FamlpsJEyaoV69e6tGjh0JDQzVt2jT5+flp9uzZ2W7/3XffqU6dOurcubNCQkLUtGlTderU6ZZHe1JTU3X58mWHDwAAYF2mLb+QlpamXbt2aejQofYxDw8PNW7cWNu2bcv2ObVr19Ynn3yi2NhY1ahRQ8ePH9c333yjLl263PTrREdH64033rjn+QHcOZbCALJnhd+N3PR7YVq5OX/+vDIyMhQUFOQwHhQUpJ9++inb53Tu3Fnnz59X3bp1ZRiG0tPT1bt371uelho6dKiioqLsn1++fFnBwcH35psAAAC5jukTiu/Ehg0bNHr0aH3wwQfavXu3li9frq+//lojR4686XO8vb0VEBDg8AEAAKzLtCM3RYsWlaenp86cOeMwfubMGRUvXjzb5wwbNkxdunTRM888I0kKDw9XcnKynn32Wb366qvy8HCprgYAAHKAaW3Ay8tL1atXV0xMjH0sMzNTMTExqlWrVrbPSUlJyVJgPD09JUmGYeRcWAAA4DJMO3IjSVFRUerWrZsiIiJUo0YNTZw4UcnJyerRo4ckqWvXripZsqSio6MlSa1bt9aECRNUrVo11axZU0ePHtWwYcPUunVre8kBAADuzdRy06FDB507d07Dhw9XYmKiqlatqlWrVtknGSckJDgcqXnttddks9n02muv6dSpUypWrJhat26tt956y6xvAQAA5DKmlhtJ6tu3r/r27ZvtYxs2bHD4PE+ePBoxYoRGjBjhhGQAAMAVMQMXAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYCuUGAABYiunlZsqUKQoJCZGPj49q1qyp2NjYW26flJSkyMhIlShRQt7e3vr3v/+tb775xklpAQBAbpfHzC++aNEiRUVFadq0aapZs6YmTpyoZs2a6dChQwoMDMyyfVpampo0aaLAwEAtXbpUJUuW1IkTJ1SwYEHnhwcAALmSqeVmwoQJ6tWrl3r06CFJmjZtmr7++mvNnj1bQ4YMybL97NmzdfHiRX333XfKmzevJCkkJMSZkQEAQC5n2mmptLQ07dq1S40bN/6/MB4eaty4sbZt25btc7744gvVqlVLkZGRCgoKUqVKlTR69GhlZGTc9Oukpqbq8uXLDh8AAMC6TCs358+fV0ZGhoKCghzGg4KClJiYmO1zjh8/rqVLlyojI0PffPONhg0bpvHjx2vUqFE3/TrR0dEqUKCA/SM4OPiefh8AACB3MX1C8Z3IzMxUYGCgpk+frurVq6tDhw569dVXNW3atJs+Z+jQobp06ZL94+TJk05MDAAAnM20OTdFixaVp6enzpw54zB+5swZFS9ePNvnlChRQnnz5pWnp6d9rGLFikpMTFRaWpq8vLyyPMfb21ve3t73NjwAAMi1TDty4+XlperVqysmJsY+lpmZqZiYGNWqVSvb59SpU0dHjx5VZmamfezw4cMqUaJEtsUGAAC4H1NPS0VFRWnGjBn66KOPdPDgQT3//PNKTk62Xz3VtWtXDR061L79888/r4sXL2rAgAE6fPiwvv76a40ePVqRkZFmfQsAACCXMfVS8A4dOujcuXMaPny4EhMTVbVqVa1atco+yTghIUEeHv/Xv4KDg7V69WoNGjRIlStXVsmSJTVgwAANHjzYrG8BAADkMqaWG0nq27ev+vbtm+1jGzZsyDJWq1Ytff/99zmcCgAAuCqXuloKAADg71BuAACApVBuAACApfyjcpOWlqZDhw4pPT39XuUBAAD4R+6q3KSkpKhnz57y8/NTWFiYEhISJEn9+vXTmDFj7mlAAACAO3FX5Wbo0KHau3evNmzYIB8fH/t448aNtWjRonsWDgAA4E7d1aXgK1as0KJFi/Twww/LZrPZx8PCwnTs2LF7Fg4AAOBO3dWRm3PnzikwMDDLeHJyskPZAQAAcLa7KjcRERH6+uuv7Z/fKDQzZ8686bpQAAAAznBXp6VGjx6tFi1a6MCBA0pPT9ekSZN04MABfffdd9q4ceO9zggAAHDb7urITd26dbV3716lp6crPDxca9asUWBgoLZt26bq1avf64wAAAC37Y6P3Fy7dk3PPfechg0bphkzZuREJgAAgLt2x0du8ubNq2XLluVEFgAAgH/srk5LtW3bVitWrLjHUQAAAP65u5pQXL58eb355pvaunWrqlevrnz58jk83r9//3sSDgAA4E7dVbmZNWuWChYsqF27dmnXrl0Oj9lsNsoNAAAwzV2Vm/j4+HudAwAA4J74R6uCS5JhGDIM415kAQAA+MfuutzMmzdP4eHh8vX1la+vrypXrqyPP/74XmYDAAC4Y3d1WmrChAkaNmyY+vbtqzp16kiStmzZot69e+v8+fMaNGjQPQ0JAABwu+6q3Lz//vuaOnWqunbtah9r06aNwsLC9Prrr1NuAACAae7qtNTp06dVu3btLOO1a9fW6dOn/3EoAACAu3VX5aZcuXJavHhxlvFFixapfPny/zgUAADA3bqr01JvvPGGOnTooE2bNtnn3GzdulUxMTHZlh4AAABnuasjN+3bt9f27dtVtGhRrVixQitWrFDRokUVGxurdu3a3euMAAAAt+2ujtxIUvXq1fXJJ5/cyywAAAD/2F0dufnmm2+0evXqLOOrV6/WypUr/3EoAACAu3VX5WbIkCHKyMjIMm4YhoYMGfKPQwEAANytuyo3R44cUWhoaJbxChUq6OjRo/84FAAAwN26q3JToEABHT9+PMv40aNHlS9fvn8cCgAA4G7dVbn573//q4EDB+rYsWP2saNHj+qFF15QmzZt7lk4AACAO3VX5WbcuHHKly+fKlSooNKlS6t06dKqUKGCihQponfeeedeZwQAALhtd3UpeIECBfTdd99p7dq12rt3r3x9fVWlShXVq1fvXucDAAC4I3d05Gbbtm366quvJEk2m01NmzZVYGCg3nnnHbVv317PPvusUlNTcyQoAADA7bijcvPmm2/qxx9/tH++f/9+9erVS02aNNGQIUP05ZdfKjo6+p6HBAAAuF13VG7i4uLUqFEj++cLFy5UjRo1NGPGDEVFRem9995jbSkAAGCqOyo3v/32m4KCguyfb9y4US1atLB//tBDD+nkyZP3Lh0AAMAduqNyExQUpPj4eElSWlqadu/erYcfftj++JUrV5Q3b957mxAAAOAO3FG5admypYYMGaLNmzdr6NCh8vPzc7hCat++fSpbtuw9DwkAAHC77uhS8JEjR+qxxx5TgwYN5O/vr48++kheXl72x2fPnq2mTZve85AAAAC3647KTdGiRbVp0yZdunRJ/v7+8vT0dHh8yZIl8vf3v6cBAQAA7sRd38QvO4ULF/5HYQAAAP6pu1p+AQAAILei3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEvJFeVmypQpCgkJkY+Pj2rWrKnY2Njbet7ChQtls9nUtm3bnA0IAABchunlZtGiRYqKitKIESO0e/duValSRc2aNdPZs2dv+byff/5ZL774ourVq+ekpAAAwBWYXm4mTJigXr16qUePHgoNDdW0adPk5+en2bNn3/Q5GRkZeuqpp/TGG2+oTJkyt3z91NRUXb582eEDAABYl6nlJi0tTbt27VLjxo3tYx4eHmrcuLG2bdt20+e9+eabCgwMVM+ePf/2a0RHR6tAgQL2j+Dg4HuSHQAA5E6mlpvz588rIyNDQUFBDuNBQUFKTEzM9jlbtmzRrFmzNGPGjNv6GkOHDtWlS5fsHydPnvzHuQEAQO6Vx+wAd+LKlSvq0qWLZsyYoaJFi97Wc7y9veXt7Z3DyQAAQG5harkpWrSoPD09debMGYfxM2fOqHjx4lm2P3bsmH7++We1bt3aPpaZmSlJypMnjw4dOqSyZcvmbGgAAJCrmXpaysvLS9WrV1dMTIx9LDMzUzExMapVq1aW7StUqKD9+/crLi7O/tGmTRs1bNhQcXFxzKcBAADmn5aKiopSt27dFBERoRo1amjixIlKTk5Wjx49JEldu3ZVyZIlFR0dLR8fH1WqVMnh+QULFpSkLOMAAMA9mV5uOnTooHPnzmn48OFKTExU1apVtWrVKvsk44SEBHl4mH7FOgAAcBGmlxtJ6tu3r/r27ZvtYxs2bLjlc+fOnXvvAwEAAJfFIREAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGAplBsAAGApuaLcTJkyRSEhIfLx8VHNmjUVGxt7021nzJihevXqqVChQipUqJAaN258y+0BAIB7Mb3cLFq0SFFRURoxYoR2796tKlWqqFmzZjp79my222/YsEGdOnXS+vXrtW3bNgUHB6tp06Y6deqUk5MDAIDcyPRyM2HCBPXq1Us9evRQaGiopk2bJj8/P82ePTvb7efPn68+ffqoatWqqlChgmbOnKnMzEzFxMQ4OTkAAMiNTC03aWlp2rVrlxo3bmwf8/DwUOPGjbVt27bbeo2UlBRdu3ZNhQsXzvbx1NRUXb582eEDAABYl6nl5vz588rIyFBQUJDDeFBQkBITE2/rNQYPHqz77rvPoSD9WXR0tAoUKGD/CA4O/se5AQBA7mX6aal/YsyYMVq4cKE+++wz+fj4ZLvN0KFDdenSJfvHyZMnnZwSAAA4Ux4zv3jRokXl6empM2fOOIyfOXNGxYsXv+Vz33nnHY0ZM0bffvutKleufNPtvL295e3tfU/yAgCA3M/UIzdeXl6qXr26w2TgG5ODa9WqddPnjRs3TiNHjtSqVasUERHhjKgAAMBFmHrkRpKioqLUrVs3RUREqEaNGpo4caKSk5PVo0cPSVLXrl1VsmRJRUdHS5LGjh2r4cOHa8GCBQoJCbHPzfH395e/v79p3wcAAMgdTC83HTp00Llz5zR8+HAlJiaqatWqWrVqlX2ScUJCgjw8/u8A09SpU5WWlqbHH3/c4XVGjBih119/3ZnRAQBALmR6uZGkvn37qm/fvtk+tmHDBofPf/7555wPBAAAXJZLXy0FAADwV5QbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKZQbAABgKbmi3EyZMkUhISHy8fFRzZo1FRsbe8vtlyxZogoVKsjHx0fh4eH65ptvnJQUAADkdqaXm0WLFikqKkojRozQ7t27VaVKFTVr1kxnz57NdvvvvvtOnTp1Us+ePbVnzx61bdtWbdu21Q8//ODk5AAAIDcyvdxMmDBBvXr1Uo8ePRQaGqpp06bJz89Ps2fPznb7SZMmqXnz5nrppZdUsWJFjRw5Ug8++KAmT57s5OQAACA3ymPmF09LS9OuXbs0dOhQ+5iHh4caN26sbdu2Zfucbdu2KSoqymGsWbNmWrFiRbbbp6amKjU11f75pUuXJEmXL1/+h+lvLTM1JUdf31ly+ufkDOyL3IN9kbtYYX+wL3KPnN4XN17fMIy/3dbUcnP+/HllZGQoKCjIYTwoKEg//fRTts9JTEzMdvvExMRst4+OjtYbb7yRZTw4OPguU7uXAhPNToAb2Be5B/si92Bf5B7O2hdXrlxRgQIFbrmNqeXGGYYOHepwpCczM1MXL15UkSJFZLPZTEz2z1y+fFnBwcE6efKkAgICzI7j1tgXuQf7IvdgX+QuVtgfhmHoypUruu+++/52W1PLTdGiReXp6akzZ844jJ85c0bFixfP9jnFixe/o+29vb3l7e3tMFawYMG7D53LBAQEuOz/qFbDvsg92Be5B/sid3H1/fF3R2xuMHVCsZeXl6pXr66YmBj7WGZmpmJiYlSrVq1sn1OrVi2H7SVp7dq1N90eAAC4F9NPS0VFRalbt26KiIhQjRo1NHHiRCUnJ6tHjx6SpK5du6pkyZKKjo6WJA0YMEANGjTQ+PHj1apVKy1cuFA7d+7U9OnTzfw2AABALmF6uenQoYPOnTun4cOHKzExUVWrVtWqVavsk4YTEhLk4fF/B5hq166tBQsW6LXXXtMrr7yi8uXLa8WKFapUqZJZ34IpvL29NWLEiCyn3OB87Ivcg32Re7Avchd32x8243auqQIAAHARpt/EDwAA4F6i3AAAAEuh3AAAAEuh3AAAAEuh3AAAAEuh3AAAYGF//PGH2RGcjnLjYtLS0nTo0CGlp6ebHQXINT7++GPVqVNH9913n06cOCFJmjhxoj7//HOTk7mfzZs363//+59q1aqlU6dOSbq+f7Zs2WJyMveSmZmpkSNHqmTJkvL399fx48clScOGDdOsWbNMTpfzKDcuIiUlRT179pSfn5/CwsKUkJAgSerXr5/GjBljcjr3xJt47jB16lRFRUWpZcuWSkpKUkZGhqTra8hNnDjR3HBuZtmyZWrWrJl8fX21Z88epaamSpIuXbqk0aNHm5zOvYwaNUpz587VuHHj5OXlZR+vVKmSZs6caWIy56DcuIihQ4dq79692rBhg3x8fOzjjRs31qJFi0xM5p54E8893n//fc2YMUOvvvqqPD097eMRERHav3+/icncz6hRozRt2jTNmDFDefPmtY/XqVNHu3fvNjGZ+5k3b56mT5+up556yuH3okqVKvrpp59MTOYclBsXsWLFCk2ePFl169aVzWazj4eFhenYsWMmJnNPvInnHvHx8apWrVqWcW9vbyUnJ5uQyH0dOnRI9evXzzJeoEABJSUlOT+QGzt16pTKlSuXZTwzM1PXrl0zIZFzUW5cxLlz5xQYGJhlPDk52aHswDl4E889Spcurbi4uCzjq1atUsWKFZ0fyI0VL15cR48ezTK+ZcsWlSlTxoRE7is0NFSbN2/OMr506dJs/xiwGtMXzsTtiYiI0Ndff61+/fpJkr3QzJw5U7Vq1TIzmlu68SYeEhLiMM6buPNFRUUpMjJSf/zxhwzDUGxsrD799FNFR0e7xdyC3KRXr14aMGCAZs+eLZvNpl9//VXbtm3Tiy++qGHDhpkdz60MHz5c3bp106lTp5SZmanly5fr0KFDmjdvnr766iuz4+U8Ay5h8+bNhr+/v9G7d2/Dx8fHGDBggNGkSRMjX758xs6dO82O53ZGjx5thIaGGt9//72RP39+Y/PmzcYnn3xiFCtWzHjvvffMjud2PvnkE6NcuXKGzWYzbDabUbJkSWPmzJlmx3I7mZmZxqhRo4x8+fLZ94WPj4/x2muvmR3NLW3atMlo3LixUaxYMcPX19eoU6eOsXr1arNjOQWrgruQ48ePKzo6Wnv37tXVq1f14IMPavDgwQoPDzc7mtsxDEOjR49WdHS0UlJSJF2f4/Hiiy9q5MiRJqdzH+np6VqwYIGaNWumoKAgpaSk6OrVq9mewkXOysjI0NatW1W5cmX5+fnp6NGjunr1qkJDQ+Xv7292PLeSnp6u0aNH6+mnn9b9999vdhxTUG5cwLVr1/Tcc89p2LBhKl26tNlx3B5v4rmLn5+fDh48qFKlSpkdxe35+Pjo4MGDvE/lAv7+/vrhhx+ynDp3F0wodgF58+bVsmXLzI6B/8/T01NNmzbVb7/9Ji8vL4WGhqpGjRoUG5PUqFFDe/bsMTsGdP0eKjduFgdzNWrUSBs3bjQ7hmmYUOwi2rZtqxUrVmjQoEFmR4H+702cv1DN16dPH73wwgv65ZdfVL16deXLl8/h8cqVK5uUzP2MGjXKfmo2u30REBBgUjL306JFCw0ZMkT79+/Pdl+0adPGpGTOwWkpFzFq1CiNHz9ejRo1yvZ/1P79+5uUzD2tWrVKQ4cO5U08F/DwyHoA2mazyTAM2Ww2+x2LkfP+vC/+fIsK9oXzZfd7cYM77AvKjYu41RECm83GoWAn400897ixltTNMBfHef7uNEiDBg2clATujnID3AXexAEg96LcuKAbu4w7EwPXHTt2TBMnTtTBgwclXb8764ABA1S2bFmTk7mfpKQkzZo1y74vwsLC9PTTT6tAgQImJ3M/Gzdu1DvvvOPwe/HSSy+pXr16JifLeVwt5ULmzZun8PBw+fr6ytfXV5UrV9bHH39sdiy3lZSUpPHjx+uZZ57RM888o3fffVeXLl0yO5bbWb16tUJDQxUbG6vKlSurcuXK2r59u8LCwrR27Vqz47mVnTt3qmzZsnr33Xd18eJFXbx4URMmTFDZsmVZc83JPvnkEzVu3Fh+fn7q37+/+vfvL19fXzVq1EgLFiwwO16O48iNi5gwYYKGDRumvn37qk6dOpKu3+p/ypQpGjVqFFdROdnOnTvtq4LXqFFDkrRjxw79/vvvWrNmjR588EGTE7qPatWqqVmzZhozZozD+JAhQ7RmzRr+UXWievXqqVy5cpoxY4by5Ll+MW56erqeeeYZHT9+XJs2bTI5ofuoWLGinn322Sz/NkyYMEEzZsywH82xKsqNiyhdurTeeOMNde3a1WH8o48+0uuvv674+HiTkrkn3sRzDx8fH+3fv1/ly5d3GD98+LAqV66sP/74w6Rk7sfX11d79uxRhQoVHMYPHDigiIgI+928kfO8vb31448/ZlkZ/OjRo6pUqZLlfy84LeUiTp8+rdq1a2cZr127tk6fPm1CIve2c+dODR482F5sJClPnjx6+eWXtXPnThOTuZ9ixYpluyp4XFwcyzA4WUBAgBISErKMnzx5Uvnz5zchkfsKDg5WTExMlvFvv/1WwcHBJiRyLm7i5yLKlSunxYsX65VXXnEYX7RoUZa/WJHzbryJ//UvVN7Ena9Xr1569tlndfz4cfsfAFu3btXYsWMVFRVlcjr30qFDB/Xs2VPvvPOOw7546aWX1KlTJ5PTuZcXXnhB/fv3V1xcnMO+mDt3riZNmmRyOidw6jKduGtLly41PD09jWbNmhlvvvmm8eabbxrNmjUz8uTJYyxfvtzseG6nX79+xv33328sXLjQSEhIMBISEoxPP/3UuP/++40BAwaYHc+tZGZmGhMmTDBKlizpsCr4xIkTjczMTLPjuZXU1FSjf//+hpeXl+Hh4WF4eHgY3t7exsCBA40//vjD7HhuZ/ny5UadOnWMwoULG4ULFzbq1KljrFixwuxYTsGcGxeya9cuvfvuu/aJYBUrVtQLL7ygatWqmZzM/aSlpemll17StGnTlJ6eLun6GmDPP/+8xowZI29vb5MTuqcrV65IEkfPTJaSkqJjx45JksqWLSs/Pz+TE8HdUG6Af4A3cfPFx8crPT09y+nZI0eOKG/evG67KrIZLl26pIyMDBUuXNhh/OLFi8qTJw/LkjjRjh07lJmZqZo1azqMb9++XZ6enoqIiDApmXMwodhFfPPNN1q9enWW8dWrV2vlypUmJHJvly5d0sWLF+Xn56fw8HCFh4fLz89PFy9e1OXLl82O51a6d++u7777Lsv49u3b1b17d+cHcmMdO3bUwoULs4wvXrxYHTt2NCGR+4qMjNTJkyezjJ86dUqRkZEmJHIuyo2LGDJkSLbrFRmGoSFDhpiQyL3xJp577Nmzx37vpz97+OGHs72KCjln+/btatiwYZbx//znP9q+fbsJidzXgQMHsr3fVrVq1XTgwAETEjkX5cZFHDlyRKGhoVnGK1SooKNHj5qQyL3xJp572Gw2+1ybP7txigTOk5qaap+D9mfXrl3T77//bkIi9+Xt7a0zZ85kGT99+rTDLSysinLjIgoUKJDtyt9Hjx5Vvnz5TEjk3ngTzz3q16+v6OhohyKTkZGh6Oho1a1b18Rk7qdGjRqaPn16lvFp06apevXqJiRyX02bNtXQoUMdloRJSkrSK6+8oiZNmpiYzDmYUOwinnvuOW3btk2fffaZfTHAo0ePqn379nrooYc0c+ZMkxO6l4YNG6pSpUp6//33HcYjIyO1b98+bd682aRk7ufAgQOqX7++ChYsaF8QcPPmzbp8+bLWrVunSpUqmZzQfWzdulWNGzfWQw89pEaNGkmSYmJitGPHDq1Zs8YtFmzMLU6dOqX69evrwoUL9itq4+LiFBQUpLVr11r+Rn6UGxdx6dIlNW/eXDt37tT9998vSfrll19Ur149LV++XAULFjQ3oJvhTTx3+fXXXzV58mTt3bvXvqhs3759s1y1g5wXFxent99+W3FxcfZ9MXToUG42aoLk5GTNnz/f4feiU6dOyps3r9nRchzlxoUYhqG1a9c6/I9av359s2O5Ld7EASB3oty4sKSkJI7YwO2tWrVK/v7+9vk1U6ZM0YwZMxQaGqopU6aoUKFCJid0H7t371bevHkVHh4uSfr88881Z84chYaG6vXXX5eXl5fJCd3HRx99pKJFi6pVq1aSpJdfflnTp09XaGioPv30U5UqVcrkhDmLCcUuYuzYsVq0aJH98yeffFJFihRRyZIltXfvXhOTuafdu3dr//799s8///xztW3bVq+88orS0tJMTOZ+XnrpJfu9hfbv36+oqCi1bNlS8fHxrC3lZM8995wOHz4sSTp+/Lg6dOggPz8/LVmyRC+//LLJ6dzL6NGj5evrK0natm2bJk+erHHjxqlo0aIaNGiQyemcwIw1H3DnQkJCjK1btxqGYRhr1qwxChYsaKxevdro2bOn0aRJE5PTuZ+IiAhj6dKlhmEYxrFjxwxvb2+jU6dORrly5Vhbysny5ctnxMfHG4ZhGCNGjDDat29vGIZh7Nq1ywgKCjIxmfsJCAgwjh49ahiGYYwZM8Zo2rSpYRiGsWXLFuP+++83M5rb8fX1NU6cOGEYhmG8/PLLRpcuXQzDMIwffvjBKFq0qJnRnIIjNy4iMTHRPrv9q6++0pNPPqmmTZvq5Zdf1o4dO0xO534OHz6sqlWrSpKWLFmiBg0aaMGCBZo7d66WLVtmbjg34+XlpZSUFEnSt99+q6ZNm0qSChcuzN2incwwDGVmZkq6vi9atmwpSQoODtb58+fNjOZ2/P39deHCBUnSmjVr7Jd/+/j4uMXtKqx/Jx+LKFSokE6ePKng4GCtWrVKo0aNknT9zYQblTnfX9/EH330UUm8iZuhbt26ioqKUp06dRQbG2s/fXv48GH7lYVwjoiICI0aNUqNGzfWxo0bNXXqVEnX1/8KCgoyOZ17adKkiZ555hlVq1ZNhw8fthfNH3/80S3WW+PIjYt47LHH1LlzZzVp0kQXLlxQixYtJF2/9Xy5cuVMTud+bryJf/zxx9q4caN90h5v4s43efJk5cmTR0uXLtXUqVNVsmRJSdLKlSvVvHlzk9O5l4kTJ2r37t3q27evXn31Vft709KlS1W7dm2T07mXKVOmqFatWjp37pyWLVumIkWKSJJ27dqlTp06mZwu53G1lIu4du2aJk2apJMnT6p79+72mzK9++67yp8/v5555hmTE7qXffv26amnnlJCQoKioqI0YsQISVK/fv104cIFLViwwOSE+KsxY8aod+/eXGFogj/++EOenp72+6t8+umnatOmDXdXzwX69OmjN998U0WLFjU7yj1FubGYVq1aaebMmSpRooTZUdwSb+K5V0BAgOLi4lSmTBmzo7g99kXuYdV9wWkpi9m0aZNbTBbLrXx8fBzu/vncc89lu3gdnI+/43IP9kXuYdV9QbkBcpBV3zgAIDej3AAAAEuh3AAAAEuh3AAAAEuh3ABwC/Xq1bOvtQNzlSpVymHiPczzv//9TwEBAWbHuOcoNy5i06ZNSk9PzzKenp6uTZs22T9/5ZVXVLhwYWdGwy3wJp7zbncR02+++YZbJOSwkydP6pdffrF/Hhsbq4EDB2r69OkO2/3www/25WSQM1atWqUtW7bYP58yZYqqVq2qzp0767fffrOPT5061XL3uJEoNy6jYcOGunjxYpbxS5cuqWHDhvbPhw4dyk3KnCgtLU2//PKLEhISHD5u4E085/11JeqOHTuyErVJOnfurPXr10u6vh5ekyZNFBsbq1dffVVvvvmmyency0svvWRfW23//v164YUX1LJlS8XHxysqKsrkdDmPcuMiDMOQzWbLMn7hwgVuEGeCI0eO2E9zlCpVSqVLl1bp0qUVEhKi0qVLmx3Prfx1EdP69euziKlJfvjhB9WoUUOStHjxYlWqVEnfffed5s+fr7lz55obzs3Ex8crNDRUkrRs2TI9+uijGj16tKZMmaKVK1eanC7nsXBmLvfYY49Jkmw2m7p37y5vb2/7YxkZGdq3bx9rtpige/fuypMnj7766iuVKFEi2+IJ52AR09zj2rVr9veob7/9Vm3atJEkVahQQadPnzYzmtvx8vJSSkqKpOv7omvXrpKkwoUL24/oWBnlJpcrUKCApOtv4Pnz53eYEOnl5aWHH35YvXr1Miue24qLi9OuXbtUoUIFs6O4PVaizj3CwsI0bdo0tWrVSmvXrtXIkSMlSb/++qt94UY4R926dRUVFaU6deooNjZWixYtknT9SOf9999vcrqcR7nJ5ebMmSNJCgkJ0YsvvsgpqFwiNDSUowK5xMSJE/XUU09pxYoVrERtsrFjx6pdu3Z6++231a1bN1WpUkWS9MUXX9hPV8E5Jk+erD59+mjp0qWaOnWqSpYsKUlauXKlmjdvbnK6nMfCmcBt+vOh3J07d+q1117T6NGjFR4enuWKKCteWulq/rqIKZwjIyNDly9fVqFChexjP//8s/z8/BQYGGhiMrgTyo0LWbp0qRYvXqyEhASHS1yl65fDImd5eHg4zK3JbpL3jbGMjAxnx3NbJ0+elM1msx9qj42N1YIFCxQaGqpnn33W5HTu5ffff5dhGPLz85MknThxQp999pkqVqyoZs2amZzOvezevVt58+ZVeHi4pOu3SJgzZ45CQ0P1+uuvy8vLy+SEOYvTUi7ivffe06uvvqru3bvr888/V48ePXTs2DHt2LFDkZGRZsdzCzcucUXu0rlzZz377LPq0qWL/fLjsLAwzZ8/X4mJiRo+fLjZEd3Gf//7Xz322GPq3bu3kpKSVLNmTeXNm1fnz5/XhAkT9Pzzz5sd0W0899xzGjJkiMLDw+23SGjXrp2WLFmilJQUTZw40eyIOcuAS3jggQeMBQsWGIZhGP7+/saxY8cMwzCMYcOGGZGRkWZGc0snTpwwMjMzs4xnZmYaJ06cMCGR+ypYsKDx008/GYZhGJMmTTJq165tGIZhrF692ihdurSZ0dxOkSJFjB9++MEwDMOYMWOGUblyZSMjI8NYvHixUaFCBZPTuZeAgADj6NGjhmEYxpgxY4ymTZsahmEYW7ZsMe6//34zozkF97lxEQkJCfbJkb6+vrpy5YokqUuXLvr000/NjOaWSpcurXPnzmUZv3jxIve5cTIuP849UlJSlD9/fknSmjVr9Nhjj8nDw0MPP/ywTpw4YXI692L85RYJLVu2lOQ+t0ig3LiI4sWL2+9Q/K9//Uvff/+9pOuXuxpMm3I64yY3Vbx69ap8fHxMSOS+blx+vHnzZq1du9Z+JQiXHztfuXLltGLFCp08eVKrV69W06ZNJUlnz55lkr2T3bhFwscff6yNGzeqVatWktznFgnMuXERjzzyiL744gtVq1ZNPXr00KBBg7R06VLt3LnTfqM/5Lwbty232WwaNmyYfeKkdP0qke3bt9vvlgvn4PLj3GP48OHq3LmzBg0apEceeUS1atWSdP0oTrVq1UxO517c/RYJXC3lIjIzM5WZmak8ea730YULF+q7775T+fLl9dxzz1l+5ntucWMdr40bN6pWrVoOP3cvLy/7/YjKly9vVkS3xOXHuUdiYqJOnz6tKlWqyMPj+smB2NhYBQQEcNPLXMBdbpFAuQHuQo8ePTRp0iQOtecS6enp2rBhg44dO6bOnTsrf/78+vXXXxUQECB/f3+z47mdo0eP6tixY6pfv758fX1vehoXOSspKUlLly7VsWPH9NJLL6lw4cLavXu3goKC7Df1syrKjQv5448/tG/fPp09e9Y+UeyGG5MoAXdz4sQJNW/eXAkJCUpNTdXhw4dVpkwZDRgwQKmpqZo2bZrZEd3GhQsX9OSTT2r9+vWy2Ww6cuSIypQpo6efflqFChXS+PHjzY7oNvbt26dGjRqpYMGC+vnnn3Xo0CGVKVNGr732mhISEjRv3jyzI+Yo5ty4iFWrVqlr167ZznLnpnHOd7N5TjabTT4+PipXrpw6d+6sBx54wMnJ3M+AAQMUERGhvXv3OkwgbteuHeuuOdmgQYOUN29eJSQkqGLFivbxDh06KCoqinLjRFFRUerRo4fGjRtnv4JNklq2bKnOnTubmMw5uFrKRfTr109PPPGETp8+bZ9/c+ODYuN8AQEBWrdunXbv3i2bzSabzaY9e/Zo3bp1Sk9P16JFi1SlShVt3brV7KiWt3nzZr322mtZ5p2FhITo1KlTJqVyT2vWrNHYsWOzLMxYvnx5LgV3sh07dui5557LMl6yZEklJiaakMi5OHLjIs6cOaOoqCi3uITPFRQvXlydO3fW5MmT7ZMmMzMzNWDAAOXPn18LFy5U7969NXjwYG3ZssXktNZ2s4L/yy+/OPzFipyXnJzscAXhDRcvXrTfiwjO4e3t7bAe3g2HDx9WsWLFTEjkXBy5cRGPP/64NmzYYHYM/H+zZs3SwIED7cVGur72VL9+/TR9+nTZbDb17dtXP/zwg4kp3UPTpk0dbiVvs9l09epVjRgxwn7jMjhHvXr1HOZy2Gw2ZWZmaty4cfYrDeEcbdq00Ztvvqlr165Jur4vEhISNHjwYLVv397kdDmPCcUuIiUlRU888YSKFSuW7SrU/fv3NymZeypUqJA++uijLBO5v/jiC3Xr1k2//fabjhw5oho1aui3334zKaV7+OWXX9SsWTMZhqEjR44oIiJCR44cUdGiRbVp0yYuBXeiH374QY0aNdKDDz6odevWqU2bNvrxxx918eJFbd26VWXLljU7otu4dOmSHn/8ce3cuVNXrlzRfffdp8TERNWqVUvffPON8uXLZ3bEHEW5cRGzZs1S79695ePjoyJFijhcVmmz2XT8+HET07mf/v3769NPP9Urr7yihx56SNL1c9yjR49W586dNWnSJM2cOVNz587ltJQTpKena+HChdq3b5+uXr2qBx98UE899ZR8fX3NjuZ2Ll26pMmTJ2vv3r32fREZGakSJUqYHc0tbdmyxeH3onHjxmZHcgrKjYsoXry4+vfvryFDhjicCoE5MjIyNGbMGE2ePFlnzpyRJAUFBalfv34aPHiwPD09lZCQIA8PjyyTKwEAOYty4yIKFy6sHTt2cFg3F7oxaY8b+pnnyJEjWr9+fbb3gBo+fLhJqdxTUlKSYmNjs90XXbt2NSmVe4qJiVFMTEy2+2L27NkmpXIOyo2LGDRokIoVK6ZXXnnF7ChArjJjxgw9//zzKlq0qIoXL57llO3u3btNTOdevvzySz311FO6evWqAgICsuyLG4v/Iue98cYbevPNNxUREaESJUpkuUP0Z599ZlIy56DcuIj+/ftr3rx5qlKliipXrpxlQvGECRNMSuaezpw5oxdffNH+V9Fff42495DzlCpVSn369NHgwYPNjuL2/v3vf6tly5YaPXp0tpeEw3lKlCihcePGqUuXLmZHMQX3uXER+/fvt6+q+9fLi1mzxfm6d++uhIQEDRs2LNu/iuA8v/32m5544gmzY0DSqVOn1L9/f4pNLpCWluYWq3/fDEduXEBGRoa2bt2q8PBwh1WPYZ78+fNr8+bNqlq1qtlR3F7Pnj310EMPqXfv3mZHcXuPPfaYOnbsqCeffNLsKG5v8ODB8vf317Bhw8yOYgqO3LgAT09PNW3aVAcPHqTc5BLBwcFZTkXBHOXKldOwYcP0/fffcw8ok7Vq1UovvfSSDhw4kO2+YIFf5/njjz80ffp0ffvtt245lYEjNy4iIiJCY8eOVaNGjcyOAl1fQ2f8+PH68MMPFRISYnYct1a6dOmbPsY9oJzrVrepYIFf57rVHaFtNpvWrVvnxDTOR7lxEatWrdLQoUM1cuRIVa9ePcvdJbkM2bkKFSqklJQUpaeny8/PL8tfRVwVAgDmody4iD//RfTnyauGYfAXkQk++uijWz7erVs3JyXBDWlpaYqPj1fZsmWVJw9n3M0wb948dejQIcsimWlpaVq4cCH3uTHB0aNHdezYMdWvX1++vr72fzOsjnLjIjZu3HjLxxs0aOCkJEDukpKSon79+tkL5+HDh1WmTBn169dPJUuW1JAhQ0xO6D48PT11+vTpLOt5XbhwQYGBgfwR5kQXLlzQk08+qfXr18tms+nIkSMqU6aMnn76aRUqVEjjx483O2KO4j7+LqJBgwa3/IDzHTt2TK+99po6deqks2fPSpJWrlypH3/80eRk7mXo0KHau3evNmzYIB8fH/t448aNtWjRIhOTuZ+bHRX45ZdfVKBAARMSua9BgwYpb968SkhIcLg0v0OHDlq1apWJyZyDY7cuJCkpSbNmzdLBgwclSWFhYXr66ad50zDBxo0b1aJFC9WpU0ebNm3SW2+9pcDAQO3du1ezZs3S0qVLzY7oNlasWKFFixbp4YcfdviHNSwsTMeOHTMxmfuoVq2abDabbDabGjVq5HBaMCMjQ/Hx8WrevLmJCd3PmjVrtHr16ixr25UvX14nTpwwKZXzUG5cxM6dO9WsWTP5+vqqRo0akq5fyvfWW29pzZo1evDBB01O6F6GDBmiUaNGKSoqSvnz57ePP/LII5o8ebKJydzPuXPnspwGkaTk5GS3mFuQG7Rt21aSFBcXp2bNmsnf39/+mJeXl0JCQtS+fXuT0rmn5OTkbG+mePHixSxzoqyIcuMiBg0apDZt2mjGjBn2v4rS09P1zDPPaODAgdq0aZPJCd3L/v37tWDBgizjgYGBOn/+vAmJ3FdERIS+/vpr9evXT9L/TbifOXOmatWqZWY0tzFixAhJUkhIiDp27OgW/3jmdvXq1dO8efM0cuRISdd/LzIzMzVu3LhbXiZuFZQbF7Fz506HYiNJefLk0csvv6yIiAgTk7mnggUL6vTp01nusbJnzx6VLFnSpFTuafTo0WrRooUOHDig9PR0TZo0SQcOHNB33333txPxcW+FhoYqLi5ONWvWdBjfvn27PD09ea9yonHjxqlRo0bauXOn0tLS9PLLL+vHH3/UxYsXtXXrVrPj5TgmFLuIgIAAJSQkZBk/efKkw2kROEfHjh01ePBgJSYm2v8i2rp1q1588UUud3WyunXrKi4uTunp6QoPD9eaNWsUGBiobdu2qXr16mbHcyuRkZE6efJklvFTp04pMjLShETuq1KlSjp8+LDq1q2r//73v0pOTtZjjz2mPXv2qGzZsmbHy3FcCu4i+vfvr88++0zvvPOOfTG0rVu36qWXXlL79u01ceJEcwO6mbS0NEVGRmru3LnKyMhQnjx5lJ6erqeeekpz586Vp6en2REBp/P399e+fftUpkwZh/H4+HhVrlxZV65cMSkZ3A2npXKxffv2qVKlSvLw8NA777wjm82mrl27Kj09XZKUN29ePf/88xozZozJSd2Pl5eXZsyYoeHDh2v//v26evWqqlWrpvLly5sdze1wb5Xcw9vbW2fOnMlSbk6fPs2NFZ1s1apV8vf3V926dSVJU6ZM0YwZMxQaGqopU6ZYfp1CjtzkYn9+0y5Tpox27NghX19f++WtZcuWzXY2PHJGVFTUbW9r9UXpchMPDw8lJiZmKTe//vqrypYtq99//92kZO6nU6dOOn36tD7//HP7LSqSkpLUtm1bBQYGavHixSYndB/h4eEaO3asWrZsqf379ysiIkIvvPCC1q9frwoVKmjOnDlmR8xRVOlcrGDBgoqPj1dgYKB+/vlnZWZmys/PT+Hh4WZHc0t79uy5re24/Ng53nvvPUnXf94zZ850uPw4IyNDmzZtUoUKFcyK55beeecd1a9fX6VKlVK1atUkXb88PCgoSB9//LHJ6dxLfHy8QkNDJUnLli1T69atNXr0aO3evVstW7Y0OV3Oo9zkYu3bt1eDBg1UokQJ2Ww2RURE3HQuBysf57z169ebHQF/8u6770q6flfcadOmOfxu3Li3yrRp08yK55ZKliypffv2af78+dq7d698fX3Vo0cPderUKcvisshZXl5eSklJkSR9++239gsdChcurMuXL5sZzSk4LZXLrVq1SkePHlX//v315ptv3vTKqAEDBjg5GZA7NGzYUMuXL7f8HALgTrRp00ZpaWmqU6eORo4cqfj4eJUsWVJr1qxR3759dfjwYbMj5ijKjYvo0aOH3nvvPS77Bm6CVcFzh48//lgffvihjh8/rm3btqlUqVJ69913VaZMGf33v/81O57bSEhIUJ8+fXTy5En1799fPXv2lHT9hrAZGRn207pWRbkB4NJ+//139e3bl1XBc4GpU6dq+PDhGjhwoEaNGqUff/xRZcqU0dy5c/XRRx9xahdOw038ALi0IUOGsCp4LvH+++9rxowZevXVVx2OnkVERGj//v0mJnMPf55Lc/ny5Vt+WB3HbgG4NFYFzz3i4+PtV0n9mbe3t5KTk01I5F4KFSpkv31IwYIFs71y0zAM2Ww2y9//iXIDwKWxKnjuUbp0acXFxalUqVIO46tWrVLFihVNSuU+1q1bp8KFC0vi6k7KDQCXxqrguUdUVJQiIyP1xx9/yDAMxcbG6tNPP1V0dLRmzpxpdjzLa9CgQbb/7Y4oNwBcGquC5x7PPPOMfH199dprryklJUWdO3fWfffdp0mTJqljx45mx3Mrc+bMkb+/v5544gmH8SVLliglJUXdunUzKZlzMKEYgEtjVfDcIT09XfPmzVPjxo115MgRXb16VYmJifrll1/slyHDeaKjo1W0aNEs44GBgRo9erQJiZyLS8EBAPeEn5+fDh48mGXODZzPx8dHP/30k0JCQhzGf/75Z1WsWNHya65xWgqAy7mTS1kDAgJyMAn+rEaNGtqzZw/lJhcIDAzUvn37spSbvXv3qkiRIuaEciLKDQCXc7PLXP/MXS55zU369OmjF154Qb/88ouqV6+ufPnyOTxeuXJlk5K5n06dOql///7Knz+/6tevL0nauHGjBgwY4BbznzgtBcDl3MlEYXe/asSZPDyyTuO02WwUTROkpaWpS5cuWrJkif2GipmZmerataumTZsmLy8vkxPmLMoNAOCeOHHixC0f53SV8x0+fNi+Qnt4eLjb7APKDQCX99tvv2nWrFk6ePCgJCk0NFQ9evSw39AMOe/atWuqUKGCvvrqK27YB9NRbgC4tE2bNql169YqUKCAIiIiJEm7du1SUlKSvvzyS/t8A+S8kiVL6ttvv6Xc5AIZGRmaO3euYmJidPbsWWVmZjo8vm7dOpOSOQflBoBLCw8PV61atTR16lR5enpKuv7G3qdPH3333Xcs2OhEo0eP1uHDhzVz5kyHhTPhfH379tXcuXPVqlUrlShRIssE/HfffdekZM5BuQHg0nx9fRUXF6cHHnjAYfzQoUOqWrWq5e/nkZu0a9dOMTEx8vf3V3h4eJarpZYvX25SMvdTtGhRzZs3Ty1btjQ7iimo1gBc2oMPPqiDBw9mKTcHDx5UlSpVTErlngoWLKj27dubHQOSvLy8VK5cObNjmIYjNwBc2qJFi/Tyyy+rX79+evjhhyVJ33//vaZMmaIxY8Y4zP/gPitwF+PHj9fx48c1efLkv70nlBVRbgC4tOzurfJn3GfF+c6dO6dDhw5Jkh544AEVK1bM5ETup127dlq/fr0KFy6ssLAw5c2b1+Fxq58i5LQUAJcWHx9vdgT8f8nJyerXr5/mzZtnvzrH09NTXbt21fvvvy8/Pz+TE7qPggULql27dmbHMA1HbgAA98Rzzz2nb7/9VpMnT1adOnUkSVu2bFH//v3VpEkTTZ061eSEcBeUGwAu79dff9WWLVuyvZ9H//79TUrlfooWLaqlS5fqP//5j8P4+vXr9eSTT+rcuXPmBHNj7nqKkNNSAFza3Llz9dxzz8nLy0tFihRxmDxps9koN06UkpKioKCgLOOBgYFKSUkxIZH7cvdThBy5AeDSgoOD1bt3bw0dOvRvJxcjZzVq1EhFihTRvHnz5OPjI0n6/fff1a1bN128eFHffvutyQndh7ufIqTcAHBpRYoUUWxsrMqWLWt2FLe3f/9+NW/eXKmpqfZ7DO3du1fe3t5as2aNwsLCTE7oPtz9FCF/5gBwaT179tSSJUvMjgFdXwrjyJEjio6OVtWqVVW1alWNGTNGR48epdg4mbufIuTIDQCXlpGRoUcffVS///67wsPDs9zPY8KECSYlcz/R0dEKCgrS008/7TA+e/ZsnTt3ToMHDzYpmftx91OETCgG4NKio6O1evVq+/ILf51QDOf58MMPtWDBgizjYWFh6tixI+XGiSZOnKjmzZvr/vvvz/YUodVx5AaASytUqJDeffddde/e3ewobs/Hx0cHDx5U6dKlHcaPHz+u0NBQ/fHHHyYlc08pKSmaP3++fvrpJ0lSxYoV9dRTT8nX19fkZDmPIzcAXJq3t7f9ahCYKzg4WFu3bs1SbrZu3ar77rvPpFTu6cYpwl69ejmMu8spQiYUA3BpAwYM0Pvvv292DEjq1auXBg4cqDlz5ujEiRM6ceKEZs+erUGDBmX5RxY568MPP1SFChWyjIeFhWnatGkmJHIujtwAcGmxsbFat26dvvrqK7dcIDA3eemll3ThwgX16dNHaWlpkq6fqho8eLCGDh1qcjr3kpiYqBIlSmQZL1asmE6fPm1CIuei3ABwaQULFtRjjz1mdgzo+gTusWPHatiwYTp48KB8fX1Vvnx5eXt7mx3N7bj7KULKDQCXNmfOHLMj4C/8/f310EMPmR3Drd04RXjt2jU98sgjkqSYmBi9/PLLeuGFF0xOl/O4WgqAJbjrAoFAdgzD0JAhQ/Tee+9lOUU4fPhwk9PlPMoNAJfm7gsEArdy9epVtzxFyNVSAFxaVFSUNm7cqC+//FJJSUlKSkrS559/ro0bN7rF4XfgVm6cIqxUqZLbFBuJIzcAXJy7LxAIICuO3ABwae6+QCCArDhyA8ClufsCgQCyotwAcGn79+9X8+bNlZqamu0CgWFhYSYnBOBslBsALs+dFwgEkBXlBoBLu7FA4NNPP+0w7i4LBALIignFAFyauy8QCCAryg0Al+buCwQCyIpyA8Cl3Vgg8K/cZYFAAFmxcCYAl+buCwQCyIoJxQBcmrsvEAggK8oNAEtw1wUCAWRFuQEAAJbChGIAAGAplBsAAGAplBsAAGAplBsAAGAplBsALmHDhg2y2WxKSkrKkdfv3r272rZtmyOvDcC5KDcAnC4xMVH9+vVTmTJl5O3treDgYLVu3VoxMTE3fU7t2rV1+vRpFShQQJI0d+5cFSxY8J5lmjRpkubOnXvPXg+AebhDMQCn+vnnn1WnTh0VLFhQb7/9tsLDw3Xt2jWtXr1akZGR+umnn7I859q1a/Ly8lLx4sXveZ6MjAzZbDZ7aQLg+jhyA8Cp+vTpI5vNptjYWLVv317//ve/FRYWpqioKH3//feSJJvNpqlTp6pNmzbKly+f3nrrLYfTUhs2bFCPHj106dIl2Ww22Ww2vf7665Kk1NRUvfjiiypZsqTy5cunmjVrasOGDfavf+OIzxdffKHQ0FB5e3srISEhy2mpVatWqW7duipYsKCKFCmiRx99VMeOHXPiTwrA3aLcAHCaixcvatWqVYqMjFS+fPmyPP7n00yvv/662rVrp/379+vpp5922K527dqaOHGiAgICdPr0aZ0+fVovvviiJKlv377atm2bFi5cqH379umJJ55Q8+bNdeTIEfvzU1JSNHbsWM2cOVM//vijAgMDs2RJTk5WVFSUdu7cqZiYGHl4eKhdu3bKzMy8Rz8NADmF01IAnObo0aMyDEMVKlT42207d+6sHj162D8/fvy4/b+9vLxUoEAB2Ww2h1NVCQkJmjNnjhISEuwrgr/44otatWqV5syZo9GjR0u6fprrgw8+UJUqVW769du3b+/w+ezZs1WsWDEdOHBAlSpVur1vGIApKDcAnOZOVnuJiIi449ffv3+/MjIy9O9//9thPDU1VUWKFLF/7uXlpcqVK9/ytY4cOaLhw4dr+/btOn/+vP2ITUJCAuUGyOUoNwCcpnz58rLZbNlOGv6r7E5b/Z2rV6/K09NTu3btkqenp8Nj/v7+9v/29fWVzWa75Wu1bt1apUqV0owZM3TfffcpMzNTlSpVsq88DiD3otwAcJrChQurWbNmmjJlivr375+lwCQlJd325d1eXl7KyMhwGKtWrZoyMjJ09uxZ1atX765zXrhwQYcOHdKMGTPsr7Nly5a7fj0AzsWEYgBONWXKFGVkZKhGjRpatmyZjhw5ooMHD+q9995TrVq1bvt1QkJCdPXqVcXExOj8+fNKSUnRv//9bz311FPq2rWrli9frvj4eMXGxio6Olpff/31bb92oUKFVKRIEU2fPl1Hjx7VunXrFBUVdTffLgATUG4AOFWZMmW0e/duNWzYUC+88IIqVaqkJk2aKCYmRlOnTr3t16ldu7Z69+6tDh06qFixYho3bpwkac6cOeratateeOEFPfDAA2rbtq127Nihf/3rX7f92h4eHlq4cKF27dqlSpUqadCgQXr77bfv+HsFYA6bcScz/AAAAHI5jtwAAABLodwAAABLodwAAABLodwAAABLodwAAABLodwAAABLodwAAABLodwAAABLodwAAABLodwAAABLodwAAABL+X/aDvBa0zdMwAAAAABJRU5ErkJggg==",
            "text/plain": [
              "<Figure size 640x480 with 1 Axes>"
            ]
          },
          "metadata": {},
          "output_type": "display_data"
        }
      ],
      "source": [
        "avg_scores = data[[\"format_score\", \"length_score\", \"completeness_score\", \"correctness_score\", \"conciseness_score\"]].mean()\n",
        "ax = avg_scores.plot.bar(title=\"Average Scores\", xlabel=\"Criteria\", ylabel=\"Score\")"
      ]
    }
  ],
  "metadata": {
    "colab": {
      "provenance": []
    },
    "kernelspec": {
      "display_name": "Python 3",
      "name": "python3"
    },
    "language_info": {
      "codemirror_mode": {
        "name": "ipython",
        "version": 3
      },
      "file_extension": ".py",
      "mimetype": "text/x-python",
      "name": "python",
      "nbconvert_exporter": "python",
      "pygments_lexer": "ipython3",
      "version": "3.10.11"
    }
  },
  "nbformat": 4,
  "nbformat_minor": 0
}
